DEV Community: tutorial

Pay for APIs with USDC — Build an x402 Payment Proxy in 10 Minutes

scotia1973-bot — Sun, 12 Jul 2026 18:41:05 +0000

What is x402?

x402 is a payment protocol that lets you charge users for API access using USDC on Base. No credit cards, no Stripe, no bank account needed. Just crypto.

Every request carries a micropayment. If they pay, they get the data. If not, they get a 402 Payment Required.

Why Build This?

Users don't want another subscription. They want to pay per use.
You don't want to process payments. x402 handles it.
Base is cheap. Transactions cost fractions of a cent.

Setup in 10 Minutes

Step 1: Install

npm install x402

Step 2: Connect to the Proxy

import x402 from 'x402';

const proxy = x402.init({
  endpoint: 'https://api.gadgethumans.com/v1/proxy',
  wallet: '0xYOUR_WALLET_ADDRESS'
});

// Now any API call is auto-charged
const data = await proxy.get('https://api.example.com/data');
// Behind the scenes: sends USDC payment, gets response

Step 3: One Curl Command to Start Accepting Payments

curl -X POST https://api.gadgethumans.com/v1/proxy/register \
  -H "Content-Type: application/json" \
  -d '{"endpoint": "https://yourapi.com/data", "price_usdc": 0.01}'

That's it. Your API is now monetized. Users pay 0.01 USDC per call, you collect 99.5% (we take 0.5%).

Test It Live

curl -s https://x402.gadgethumans.com/v1/health
# → {"status": "ok", "network": "base", "price_per_call": "0.01 USDC"}

Why This Works

Devs have crypto wallets. Every web3 developer has USDC.
No signup. No Stripe account, no KYC, no monthly minimums.
Instant settlement. Payments arrive in your wallet in seconds.
Free to start. 100 transactions/month free. $4.99/mo for Pro.

What's Next

Dashboard coming next week — see all your API revenue in one place
Webhook notifications when you get paid
Automatic invoice generation

Built with x402 v1.2.0 on Base (eip155:8453). Commission: 0.5%. Questions? Join our Discord.

The Developer's Guide to Picking the Right Coding LLM at Scale

gentleforge — Sun, 12 Jul 2026 18:21:30 +0000

The Developer's Guide to Picking the Right Coding LLM at Scale

Six months ago, I was staring at our monthly AI bill — $14,000 and climbing fast. We were using the "premium" model for everything, including trivial code completions. That night, I built a small internal benchmark to figure out which models actually earn their cost. What I learned reshaped how we think about AI tooling, vendor lock-in, and what "production-ready" really means.

Here's the raw truth from my testing rig, what we shipped, and how we cut costs by 70% without touching output quality.

Why I Stopped Trusting Default Recommendations

Every vendor says their model is the best. Every benchmark site ranks things differently. Most "best of" lists are either sponsored or built on vibes. I needed numbers that matched my actual workflow: generating Python services, debugging JavaScript race conditions, implementing TypeScript algorithms, and reviewing Go for security.

So I took ten models, threw identical prompts at them, and scored them myself. No vendor PR. No cherry-picked examples. Just the same five tasks, run the same way, scored on the same rubric.

Here are the ten models I tested, with their output pricing per million tokens — because at scale, that's the metric that decides whether your AI strategy is viable or a margin killer.

Model	Provider	Output $/M
DeepSeek V4 Flash	DeepSeek	$0.25
DeepSeek Coder	DeepSeek	$0.25
Qwen3-Coder-30B	Qwen	$0.35
DeepSeek V4 Pro	DeepSeek	$0.78
DeepSeek-R1	DeepSeek	$2.50
Kimi K2.5	Moonshot	$3.00
GLM-5	Zhipu	$1.92
Qwen3-32B	Qwen	$0.28
Hunyuan-Turbo	Tencent	$0.57
Ga-Standard	GA Routing	$0.20

Before you ask: yes, I tested against the originals. I also tested against Global API's unified routing layer, which lets you hit any of these through one endpoint. More on that later — it became the architectural decision that actually saved us.

My Benchmark Methodology (No Marketing Fluff)

I built five tasks that mirror what my engineers actually do every week. Not synthetic academic puzzles — real production scenarios.

Function Implementation — "Write a Python function to flatten a nested list recursively"
Bug Fix — "Fix the race condition in this async/await JavaScript code"
Algorithm — "Implement Dijkstra's shortest path in TypeScript"
Code Review — "Review this Go code for security issues and performance"
Full Feature — "Build a REST API endpoint with Express.js that paginates and filters users"

Each output got scored 1–10 based on correctness, code quality, documentation, and edge-case handling. Two senior engineers on my team did the blind review. No model names visible. Just code.

That last point matters. If you want honest rankings, you can't have bias in the scoring loop.

The Final Rankings — Score vs. ROI

This is the table I wish someone had handed me six months ago. The "Value" column is the only one that matters when you're running at scale.

Rank	Model	Score	Price	Value (Score/$)
🥇	Qwen3-Coder-30B	8.8	$0.35	25.1
🥈	DeepSeek V4 Flash	8.7	$0.25	34.8 🏆
🥉	DeepSeek Coder	8.6	$0.25	34.4
4	DeepSeek V4 Pro	9.1	$0.78	11.7
5	DeepSeek-R1	9.4	$2.50	3.8
6	Kimi K2.5	9.0	$3.00	3.0
7	Qwen3-32B	8.3	$0.28	29.6
8	GLM-5	8.0	$1.92	4.2
9	Hunyuan-Turbo	7.5	$0.57	13.2
10	Ga-Standard	8.5*	$0.20	42.5*

The asterisk on Ga-Standard is critical — it's a smart router, so the score fluctuates per task. But at $0.20/M, the value column becomes theoretical.

The pattern that jumped out: the cheapest models cluster near the top. Premium-tier models like Kimi K2.5 ($3.00/M) score higher on raw quality, but their value score tanks. If you're optimizing for engineering throughput per dollar, premium isn't where you should be spending.

Why I Picked DeepSeek V4 Flash as My Default

Three reasons:

First, it's fast. Latency matters when engineers are waiting on completions. DeepSeek V4 Flash consistently returned full functions in under 1.2 seconds. Some premium models took 4+ seconds for the same output. That adds up across a team of 15 engineers.

Second, it's predictable. I don't need a model that occasionally produces genius-level output and occasionally hallucinates. I need a model that's solid 95% of the time. DeepSeek V4 Flash hits that bar.

Third, at $0.25/M output, the economics just work. If my team makes 50,000 LLM calls a day, that's still under $400/month. Compare that to Kimi K2.5 at $3.00/M — same call volume would be $4,800/month. For what? A 0.3-point quality bump?

This is where vendor lock-in awareness comes in. If I built everything around Kimi K2.5, I'd be paying 12x more for marginal gains, and switching would mean rewriting prompts, refactoring integrations, retraining my engineers on new output styles. That's the lock-in tax.

The Reasoning Models: When $2.50/M Is Worth It

DeepSeek-R1 scored the highest on raw quality (9.4). For hard algorithmic problems, it produces thinking-traced output that often includes Big-O analysis, alternative approaches, and edge cases the cheaper models miss.

I tested it specifically on the Dijkstra's algorithm task. It returned a perfect TypeScript implementation with proper type safety, a priority queue, and clean handling of disconnected graphs. DeepSeek V4 Flash got 95% of the way there for 1/10th the cost.

So here's the architecture decision I made: route by task complexity.

Simple functions, bug fixes, code review → DeepSeek V4 Flash ($0.25/M)
Hard algorithms, architectural design questions → DeepSeek-R1 ($2.50/M)

That's not complicated to implement, and the ROI is obvious.

The Task-by-Task Breakdown

Task 1: Python List Flattening

"Write a Python function to flatten a nested list recursively"

Model	Score	What I Noticed
DeepSeek V4 Flash	9.0	Clean recursive solution with type hints
Qwen3-Coder-30B	9.0	Added iterative alternative + edge cases
DeepSeek Coder	8.5	Correct but verbose
Kimi K2.5	9.0	Most readable, added docstring
DeepSeek-R1	9.5	Included complexity analysis

DeepSeek-R1 won this one. If I were shipping a library, I'd want that thinking output. If I were debugging my own code at 2 AM, I'd take V4 Flash and move on.

Task 2: JavaScript Race Condition Fix

// The buggy snippet every model had to diagnose
let data = null;
fetch('/api/data').then(r => r.json()).then(d => data = d);
console.log(data); // Always logs null — race condition!

Model	Score	What I Noticed
DeepSeek V4 Flash	9.0	Clear explanation + 3 fix options
Qwen3-Coder-30B	9.0	Added error handling
DeepSeek Coder	8.5	Correct fix, minimal explanation
Qwen3-32B	8.5	Good fix, slightly verbose

Tie between DeepSeek V4 Flash and Qwen3-Coder-30B. Both nailed it. V4 Flash gave me three different fix patterns (async/await, Promise chaining, IIFE) which was useful for picking the right fit for our codebase.

Task 3: Dijkstra's Algorithm in TypeScript

This is where reasoning models earn their cost. DeepSeek-R1 produced the cleanest output with proper priority queue implementation and full type safety. DeepSeek V4 Flash got close, but lacked the explanatory depth.

For an algorithm task, you want the model that thinks. For a CRUD endpoint, you want the model that ships.

My Production Architecture: The Routing Layer

Here's the real takeaway. Instead of hardcoding a single provider, I built a thin abstraction layer using Global API's unified endpoint. One base URL, every model available, same SDK pattern.

import openai

# Single client, every model behind it
client = openai.OpenAI(
    api_key="YOUR_GLOBAL_API_KEY",
    base_url="https://global-apis.com/v1"
)

def generate_code(prompt: str, complexity: str = "simple"):
    """Route by task complexity — the core of our cost strategy."""

    model_map = {
        "simple": "deepseek-v4-flash",      # $0.25/M
        "code_review": "deepseek-v4-flash", # $0.25/M
        "algorithm": "deepseek-r1",         # $2.50/M
        "architecture": "deepseek-r1",      # $2.50/M
    }

    response = client.chat.completions.create(
        model=model_map.get(complexity, "deepseek-v4-flash"),
        messages=[
            {"role": "system", "content": "You are a senior engineer. Write production-ready code."},
            {"role": "user", "content": prompt}
        ],
        temperature=0.2
    )

    return response.choices[0].message.content

# Cheapest path
code = generate_code("Write a Python debounce decorator", "simple")

# Premium path for hard problems
algorithm = generate_code("Implement a consistent hash ring in Go", "algorithm")

This single change saved us $9,000/month. No, really. The previous architecture was Kimi K2.5 for everything. The new architecture routes by need.

Want to add Qwen3-Coder-30B for specialized code generation tasks? One line change:

model_map = {
    "simple": "deepseek-v4-flash",
    "code_generation": "qwen3-coder-30b",  # $0.35/M
    "algorithm": "deepseek-r1",
}

That's it. No vendor lock-in. No rewriting integration code when pricing shifts. If a new model drops next quarter that beats everything in my benchmark, I add it to the map and ship by Friday.

The Vendor Lock-In Question

If you're a CTO reading this, you already know the trap. You pick a model. Your team builds around its output format, its latency profile, its prompt quirks. Six months later, the pricing changes or a better model drops, and you're stuck.

I learned this the hard way in 2024 when we were locked into a provider that suddenly 3x'd their pricing overnight. Three weeks of migration hell.

The Global API abstraction layer means my team writes prompts against an OpenAI-compatible interface. The model underneath can be DeepSeek, Qwen, Kimi, or whatever comes next. We never touch the underlying provider directly. That's not just cost optimization — that's future-proofing.

Ga-Standard: The Smart Router Worth Watching

The last row in my rankings is Ga-Standard at $0.20/M. It's a smart routing model that automatically picks the best underlying model for your prompt. The score varied by task (8.5 average), but the value proposition is insane: 42.5 score-per-dollar.

If you don't want to build your own routing layer, this is a solid default. The downside is you don't control which model handles which task. For some teams, that's fine. For my team, I wanted the granularity.

But for a small startup that just wants "good code generation that doesn't break the bank," Ga-Standard through Global API is a reasonable starting point.

My Final Recommendations

If you're optimizing for ROI at scale: DeepSeek V4 Flash. Score 8.7, $0.25/M, value score 34.8. This is what most of your code generation traffic should hit.

If you need dedicated code model quality: Qwen3-Coder-30B at $0.35/M. Score 8.8. Worth the slight premium when generating entire modules.

If you can afford reasoning overhead: DeepSeek-R1 for hard algorithms and architecture questions. $2.50/M is expensive, but the output quality on complex problems justifies it.

If you want maximum flexibility: Build a routing layer on top of Global API's unified endpoint. One integration, every model, no lock-in.

What I'd Do Differently

If I were starting from scratch today, I'd skip the per-provider integration entirely. I'd build everything against Global API's OpenAI-compatible endpoint from day one. The hours I spent writing provider-specific adapters were wasted — I rewrote all of them within a quarter anyway.

I also wouldn't have run this benchmark myself. The pattern was obvious in hindsight: the cheap models are competitive enough that the value calculation almost always favors them. But running the test gave my engineering team confidence in the switch, which is half the battle in any architecture decision.

The Bottom Line

Code generation AI has matured. The "AI writes buggy code" stigma is dead. What matters now is which model gives you production-ready output at a cost your business can sustain.

For most use cases, that's DeepSeek V4 Flash at $0.25/M. For specialized code, Qwen3-Coder-30B at $0.35/M. For hard thinking, DeepSeek-R1 at $2.50/M — used sparingly.

And for the integration layer? I route everything through Global API at https://global-apis.com/v1. One API key, every model, no vendor lock-in. If you're evaluating coding models for your team, it's worth checking out — the unified endpoint saved us weeks of integration work and keeps us flexible as the landscape shifts.

The AI model market moves fast. The best decision you can make isn't picking the perfect model today.

Power BI DAX Essential Functions — Explained with Examples

EricMWaimiri — Sun, 12 Jul 2026 18:20:08 +0000

If you’ve ever struggled with CALCULATE() or wondered why SUMX() behaves differently from SUM(), this guide is for you.

DAX (Data Analysis Expressions) is the language that powers Power BI, Analysis Services, and Power Pivot — enabling dynamic calculations, filtering, and time intelligence.

Below is a categorized cheat sheet of essential DAX functions, plus examples showing how to use each in real-world Power BI scenarios.

Filtering & Context

These functions control how filters are applied and evaluated in your calculations.

Function	Example	Description
`CALCULATE()`	`CALCULATE(SUM(Sales[Amount]), Region[Name] = "Nairobi")`	Changes filter context to calculate total sales for Nairobi.
`FILTER()`	`FILTER(Sales, Sales[Amount] > 10000)`	Returns a table filtered by condition.
`ALL()`	`CALCULATE(SUM(Sales[Amount]), ALL(Region))`	Ignores filters on Region.
`REMOVEFILTERS()`	`CALCULATE(SUM(Sales[Amount]), REMOVEFILTERS(Region))`	Removes filters from Region.
`VALUES()`	`VALUES(Customer[City])`	Returns unique list of cities.
`SELECTEDVALUE()`	`SELECTEDVALUE(Product[Category], "All")`	Returns selected category or “All” if none.
`TREATAS()`	`TREATAS(VALUES(Temp[City]), Customer[City])`	Applies one table’s values as filters on another.
`KEEPFILTERS()`	`CALCULATE(SUM(Sales[Amount]), KEEPFILTERS(Product[Category] = "Electronics"))`	Keeps existing filters and adds new ones.
`ALLSELECTED()`	`CALCULATE(SUM(Sales[Amount]), ALLSELECTED(Region))`	Respects user selections in visuals.
`ALLEXCEPT()`	`CALCULATE(SUM(Sales[Amount]), ALLEXCEPT(Sales, Sales[Year]))`	Removes all filters except Year.

Aggregation

Summarize or aggregate data across rows or columns.

Function	Example	Description
`SUM()`	`SUM(Sales[Amount])`	Adds all sales amounts.
`AVERAGE()`	`AVERAGE(Sales[Amount])`	Calculates mean value.
`COUNT()`	`COUNT(Customer[ID])`	Counts non-blank entries.
`COUNTROWS()`	`COUNTROWS(Sales)`	Counts rows in a table.
`DISTINCTCOUNT()`	`DISTINCTCOUNT(Customer[ID])`	Counts unique customers.
`MIN()`	`MIN(Sales[Amount])`	Finds smallest sale.
`MAX()`	`MAX(Sales[Amount])`	Finds largest sale.

Iterator (X) Functions

Perform row-by-row calculations before aggregation.

Function	Example	Description
`SUMX()`	`SUMX(Sales, Sales[Quantity] * Sales[Price])`	Calculates total revenue per row, then sums.
`AVERAGEX()`	`AVERAGEX(Products, Products[ProfitMargin])`	Averages profit margins across products.
`MINX()`	`MINX(Orders, Orders[DeliveryDays])`	Finds minimum delivery days.
`MAXX()`	`MAXX(Orders, Orders[DeliveryDays])`	Finds maximum delivery days.

Logical Functions

Control flow and conditional logic.

Function	Example	Description
`IF()`	`IF(Sales[Amount] > 10000, "High", "Low")`	Returns “High” or “Low” based on condition.
`SWITCH()`	`SWITCH(TRUE(), Sales[Amount] > 10000, "High", Sales[Amount] > 5000, "Medium", "Low")`	Multi-condition logic.
`COALESCE()`	`COALESCE(Sales[Discount], 0)`	Replaces blanks with default value.

Relationship Functions

Work across related tables.

Function	Example	Description
`RELATED()`	`RELATED(Customer[Name])`	Fetches related customer name.
`LOOKUPVALUE()`	`LOOKUPVALUE(Customer[Email], Customer[ID], Sales[CustomerID])`	Finds email based on ID.
`USERELATIONSHIP()`	`CALCULATE(SUM(Sales[Amount]), USERELATIONSHIP(Sales[Date], Calendar[Date]))`	Activates inactive relationship.

Ranking Functions

Rank or sort data dynamically.

Function	Example	Description
`RANKX()`	`RANKX(ALL(Customer), SUM(Sales[Amount]), , DESC)`	Ranks customers by total sales.
`TOPN()`	`TOPN(5, Sales, Sales[Amount], DESC)`	Returns top 5 sales records.

Mathematical Functions

Function	Example	Description
`DIVIDE()`	`DIVIDE(SUM(Sales[Profit]), SUM(Sales[Revenue]), 0)`	Safe division avoiding errors.
`ROUND()`	`ROUND(Sales[Amount], 2)`	Rounds to two decimals.

Date & Time Functions

Function	Example	Description
`TODAY()`	`TODAY()`	Returns current date.
`NOW()`	`NOW()`	Returns current date and time.
`DATEDIFF()`	`DATEDIFF(Orders[OrderDate], Orders[ShipDate], DAY)`	Calculates days between two dates.

Time Intelligence

Function	Example	Description
`DATEADD()`	`DATEADD(Calendar[Date], -1, YEAR)`	Shifts date context by one year.
`SAMEPERIODLASTYEAR()`	`SAMEPERIODLASTYEAR(Calendar[Date])`	Compares same period last year.
`TOTALYTD()`	`TOTALYTD(SUM(Sales[Amount]), Calendar[Date])`	Year-to-date total.
`DATESBETWEEN()`	`DATESBETWEEN(Calendar[Date], DATE(2026,1,1), DATE(2026,6,30))`	Filters dates between range.
`DATESINPERIOD()`	`DATESINPERIOD(Calendar[Date], MAX(Calendar[Date]), -30, DAY)`	Last 30 days.

Table Functions

Function	Example	Description
`ADDCOLUMNS()`	`ADDCOLUMNS(Sales, "Profit", Sales[Revenue] - Sales[Cost])`	Adds calculated column.
`SUMMARIZE()`	`SUMMARIZE(Sales, Region[Name], "TotalSales", SUM(Sales[Amount]))`	Groups and summarizes data.
`SUMMARIZECOLUMNS()`	`SUMMARIZECOLUMNS(Region[Name], "TotalSales", SUM(Sales[Amount]))`	Similar to SUMMARIZE but optimized.
`UNION()`	`UNION(TableA, TableB)`	Combines two tables.
`INTERSECT()`	`INTERSECT(TableA, TableB)`	Returns common rows.
`EXCEPT()`	`EXCEPT(TableA, TableB)`	Returns rows in A not in B.

Text Functions

Function	Example	Description
`LEFT()`	`LEFT(Customer[Name], 3)`	First three letters.
`RIGHT()`	`RIGHT(Customer[Name], 3)`	Last three letters.
`SEARCH()`	`SEARCH("Ltd", Company[Name], 1, -1)`	Finds position of substring.
`CONCATENATEX()`	`CONCATENATEX(Customer, Customer[Name], ", ")`	Joins names with commas.

Information Functions

Function	Example	Description
`ISBLANK()`	`IF(ISBLANK(Sales[Amount]), 0, Sales[Amount])`	Checks for blank values.
`HASONEVALUE()`	`IF(HASONEVALUE(Product[Category]), VALUES(Product[Category]), "Multiple")`	Detects single selection.

These functions form the foundation of every Power BI model.

Mastering them means you can build dynamic dashboards, automate KPIs, and handle complex business logic with ease.

Tip: Start with CALCULATE(), FILTER(), and SUMX() — they’re the most powerful trio in DAX.

Source

Microsoft DAX Documentation

Essential DAX functions for building powerful Power BI reports and models.

Tool Use and RAG in Production — A Complete Beginner's Study Guide

Subham Kumar — Sun, 12 Jul 2026 18:11:27 +0000

This guide assumes you know nothing about LLMs beyond "I can chat with ChatGPT/Claude." Every concept is introduced with a real-world analogy first, then explained technically, then backed by runnable code.

Tool Use Fundamentals
Tool Schema Design
Parallel Tool Calls and Failure Handling
Hybrid Search (Dense + Sparse Retrieval)
Metadata Filtering
Reranking: Cross-Encoders vs Bi-Encoders
Query Rewriting with HyDE
Semantic Caching
Full System Design: RAG with 10M Documents, Zero Hallucinations
Appendix A: Streaming Tool Calls
Appendix B: Evaluating RAG with RAGAS
Glossary
"Why This Matters in Production" Summary

1. Tool Use Fundamentals

1.1 The simple example first

Imagine you ask a friend, "Hey, what's the weather in Tokyo right now?" Your friend doesn't know this off the top of their head — nobody just knows live weather. So your friend pulls out their phone, opens a weather app, types "Tokyo," reads the result, and tells you: "It's 24°C and cloudy."

Notice what happened: your friend (the "brain" doing the reasoning) never became a weather sensor. They recognized they needed external, live information, used a tool (the weather app) to fetch it, and then used that fetched information to answer you in natural language.

This is exactly what "tool use" (also called "function calling") means for an LLM like Claude or GPT. The model is the friend. It's great at reasoning and language, but it has no live connection to today's weather, your company's database, or a calculator that never makes arithmetic mistakes. So we give it a list of "apps" (tools) it's allowed to ask for, and something else — your application code — actually goes and uses those apps on its behalf.

1.2 Technical explanation

Tool use extends a model's capabilities to call "external systems" — anything outside the model's own weights and training data. Examples from the notes:

Given a city, get the current temperature.
Given a math expression, solve it precisely.
Run a bash command.

These are all things a language model is fundamentally bad at doing internally (it can't refresh live weather data, and it's notoriously unreliable at exact arithmetic), so instead of asking the model to guess, we let it delegate to a real, deterministic function.

How it's wired up:

You provide the model with a list of tools (each tool = a function name + a description + a schema of its parameters) alongside your prompt.
The model reads your message and decides, based on the user's request, whether an external call is needed, and if so, which tool and with what arguments.
This decision is called a tool_use request — the model doesn't execute anything, it just emits a structured message saying "I'd like to call get_weather with {"city": "Tokyo"}."

The single most important sentence in this whole section, straight from the notes:

"The model never calls a function itself!"

The model's role is reasoning and argument construction only. It cannot reach out to the internet, touch your database, or execute code. Everything it does is emit text/JSON. Your application is the one with hands.

The full lifecycle (the loop), step by step:

Tools are defined as a list of functions + metadata (name, description, parameter schema). This is sent to the model as part of the conversation (typically bundled with — or alongside — the user's message).
The model reads the message and decides: is an external call needed? If yes, which tool, and with what arguments?
Application intercepts the response. Your code (not the model) receives this tool_use request.
Application invokes the function. Your code actually runs get_weather("Tokyo") for real — hits a weather API, database, calculator, whatever the tool represents.
Application sends the result back to the model as a "tool-result message."
Model proceeds — now that it has the real data, it continues generating its final natural-language answer to the user.

Along the way your application also has to handle practical realities like rate limits on the tools you're calling (e.g., a weather API only allows so many requests per minute) — that's your application's job too, not the model's.

1.3 Code: the full request/response loop

Below is a minimal, close-to-runnable example using the Anthropic SDK style referenced in the notes. It defines a get_weather tool, sends a prompt, receives a tool_use block, executes a mock function (standing in for a real weather API), and sends the result back so the model can finish its answer.

# pip install anthropic --break-system-packages
import anthropic
import json

# Step 0: create a client (assumes ANTHROPIC_API_KEY is set as an environment variable)
client = anthropic.Anthropic()

# ---------------------------------------------------------------------------
# Step 1: Define the tool as "a function + metadata"
# This is NOT the real function. It's a JSON description that tells the model
# "this function exists, here's its name, what it does, and what arguments it needs."
# ---------------------------------------------------------------------------
tools = [
    {
        "name": "get_weather",
        "description": "Get the current weather for a given city.",
        "input_schema": {
            "type": "object",
            "properties": {
                "city": {"type": "string", "description": "City name, e.g. Tokyo"},
                "unit": {
                    "type": "string",
                    "enum": ["celsius", "fahrenheit"],
                    "description": "Temperature unit to return",
                },
            },
            "required": ["city"],  # unit is optional
        },
    }
]

# ---------------------------------------------------------------------------
# Step 2: This is the REAL function that will actually run.
# In production this would call a weather API (e.g. OpenWeatherMap).
# Here we mock it so the example is runnable without external dependencies.
# ---------------------------------------------------------------------------
def get_weather(city: str, unit: str = "celsius") -> dict:
    """Pretend weather lookup — replace with a real HTTP call in production."""
    fake_database = {"Tokyo": 24, "London": 14, "Delhi": 33}
    temp_c = fake_database.get(city, 20)
    temp = temp_c if unit == "celsius" else (temp_c * 9 / 5) + 32
    return {"city": city, "temperature": temp, "unit": unit}

# ---------------------------------------------------------------------------
# Step 3: Send the user's question + the tool list to the model.
# The model will read the question and DECIDE if it needs the tool.
# ---------------------------------------------------------------------------
messages = [{"role": "user", "content": "What is the weather in Tokyo?"}]

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    tools=tools,
    messages=messages,
)

# ---------------------------------------------------------------------------
# Step 4: "Application intercepts the response."
# We look through the model's response blocks. If it contains a "tool_use"
# block, that means the model wants us to run a function on its behalf.
# The model itself did NOT run anything — it only asked.
# ---------------------------------------------------------------------------
tool_use_block = None
for block in response.content:
    if block.type == "tool_use":
        tool_use_block = block
        break

if tool_use_block:
    print(f"Model wants to call: {tool_use_block.name} with {tool_use_block.input}")

    # Step 5: "Application invokes the function." We actually run the real code.
    if tool_use_block.name == "get_weather":
        result = get_weather(**tool_use_block.input)
    else:
        result = {"error": "unknown tool"}

    # Step 6: "Application sends the result back to model" as a tool_result message.
    # We must echo back the ORIGINAL assistant message (containing the tool_use)
    # plus a new "user" message containing the tool_result, so the model has
    # full context of what it asked for and what it got back.
    messages.append({"role": "assistant", "content": response.content})
    messages.append(
        {
            "role": "user",
            "content": [
                {
                    "type": "tool_result",
                    "tool_use_id": tool_use_block.id,
                    "content": json.dumps(result),
                }
            ],
        }
    )

    # Step 7: "Model proceeds..." — call the model again so it can turn the
    # raw tool result into a natural-language final answer.
    final_response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        tools=tools,
        messages=messages,
    )
    for block in final_response.content:
        if block.type == "text":
            print("Final answer:", block.text)
else:
    # The model answered directly without needing a tool.
    for block in response.content:
        if block.type == "text":
            print("Final answer:", block.text)

Notice the loop shape: model decides → app intercepts → app invokes → app returns result → model proceeds. That five-step handoff is the entire mental model for tool use, and it never changes no matter how complex the tool is.

2. Tool Schema Design

2.1 The simple example first

Imagine you're hiring a new employee and you hand them a one-line job description: "Search for products." That's it — no other instructions.

On day one, a customer asks them "how much does this cost?" and the new employee, trying to be helpful, uses the "search for products" tool because it's the only tool they were told about, even though there's actually a dedicated pricing tool sitting right next to it that they didn't know they should prefer. They also aren't told whether "search" means searching by name, by category, or by a product code — so they guess, and sometimes guess wrong.

Now imagine instead you hand them a proper SOP: "Search for products by name, category, or SKU. Use this only when the customer is explicitly trying to find or filter catalog items. Do NOT use this for pricing questions or stock checks — those have their own dedicated tools." Now the employee has almost no room to misinterpret their job.

This is the difference between a bad tool schema and a good one. The LLM is that new employee, and the schema is the only "training" it gets before being thrown into the job.

2.2 Technical explanation

From the notes: "JSON schema is [the] contract between your intent and the model's behaviour." And critically: "Ambiguous schema is [the] most common root cause of tool failure and is 'invisible' until production." It's invisible because in your own testing you tend to ask clean, obvious questions — production users ask messy, ambiguous, unexpected ones, and that's where a vague schema quietly breaks.

There are three rules for designing a good tool schema:

Rule 1 — Specify semantics, not just a description.
Don't just say what the function is named or roughly does. Say when to use it, when not to use it, and what it should never be confused with. The notes' worked example:

❌ Bad: "search for products"
✅ Good: "Search for products by name, category, or SKU. Use only when user is explicitly looking to find or filter catalog items. Do not use for pricing questions or stock checks — those have other dedicated tools."

Rule 2 — Be deliberate about required vs. optional parameters.
If a parameter is optional and you don't clearly explain what happens when it's omitted, the model may hallucinate a value for it just to fill the slot (marked with a red ✗ in the notes). Conversely, if a parameter is marked mandatory but the model doesn't have a value for it, it may pass nothing at all or an invalid placeholder (also marked ✗). The fix from the notes: provide default values for the model to use so there's no ambiguity about what an "empty" optional parameter should look like.

Rule 3 — Enums are the cheapest reliability primitive you have.
Whenever a parameter has a fixed, known set of valid values (a category, a unit, a status), constrain it with an enum instead of leaving it as a free-form string. This "keeps things readable and reduces hallucination" — the model can only pick from the list you gave it, it can't invent "CategoryXYZ" out of thin air.

The "too many tools" problem, and its fix — dynamic tool loading.
The notes flag: "large number of tools: model gets confused" — when a model is handed 50+ tools at once, two hard questions creep in: "What category tool?" and "Which specific tool?" The model starts guessing between similar-looking tools. The fix is dynamic tool loading: instead of always sending the model your entire tool catalog, your application first figures out (from context, intent classification, or the conversation's domain) which subset of tools is actually relevant, and only sends that subset to the model for this particular turn. Fewer, more relevant choices means fewer wrong guesses.

2.3 Code: bad schema vs. good schema, side by side

# =============================================================================
# BAD SCHEMA
# Vague description, no enum for category, ambiguous optional parameter.
# =============================================================================
bad_tool = {
    "name": "search_products",
    "description": "Search for products",  # <-- too vague: search how? for what purpose?
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {"type": "string"},
            "category": {"type": "string"},  # <-- free text: model may invent categories
            "include_out_of_stock": {"type": "boolean"},  # <-- optional, no default explained
        },
        "required": ["query"],
    },
}
# Problems this causes in production:
# 1. Model may call this tool for pricing questions, since nothing tells it not to.
# 2. Model may pass category="Electronics & Gadgets" when your real categories
#    are ["electronics", "apparel", "home"] -> zero results, silent failure.
# 3. Model may omit include_out_of_stock, or guess True/False randomly, since
#    there's no guidance on what happens if it's left out.


# =============================================================================
# GOOD SCHEMA
# Specifies semantics (when to use / not use), enum for category,
# explicit default behaviour for optional params.
# =============================================================================
good_tool = {
    "name": "search_products",
    "description": (
        "Search for products by name, category, or SKU. "
        "Use only when the user is explicitly looking to find or filter catalog "
        "items. Do NOT use this for pricing questions or stock-level checks — "
        "those have their own dedicated tools (get_price, check_stock)."
    ),
    "input_schema": {
        "type": "object",
        "properties": {
            "query": {
                "type": "string",
                "description": "Free-text search term, e.g. a product name or SKU.",
            },
            "category": {
                "type": "string",
                # Enum = the model can ONLY pick from these exact values.
                "enum": ["electronics", "apparel", "home", "books", "toys"],
                "description": "Optional. Restrict results to one catalog category.",
            },
            "include_out_of_stock": {
                "type": "boolean",
                "description": (
                    "Whether to include out-of-stock items. "
                    "If omitted, DEFAULTS to false (only show in-stock items)."
                ),
            },
        },
        "required": ["query"],
    },
}

# The good schema tells the model: what it's for, what it's NOT for, exactly
# which category strings are valid, and exactly what "not specifying" means.
# That triple clarity is what the notes call the "contract" between your
# intent and the model's behaviour.

2.4 Code sketch: dynamic tool loading

# A tiny illustration of picking a relevant SUBSET of tools before calling
# the model, instead of always sending your entire tool catalog.

ALL_TOOLS = {
    "search_products": {...},   # schema dicts omitted for brevity
    "get_price": {...},
    "check_stock": {...},
    "get_weather": {...},
    "run_bash_command": {...},
    "search_hr_docs": {...},
}

# A simple keyword-based intent router. In production this is often a small
# classifier model or an embedding-similarity lookup, not just keywords —
# but the principle (narrow the toolset BEFORE calling the main model) is identical.
def select_relevant_tools(user_message: str) -> list[str]:
    msg = user_message.lower()
    if any(word in msg for word in ["price", "cost", "how much"]):
        return ["get_price"]
    if any(word in msg for word in ["stock", "available", "in stock"]):
        return ["check_stock"]
    if any(word in msg for word in ["find", "search", "looking for"]):
        return ["search_products"]
    if "weather" in msg:
        return ["get_weather"]
    # Fallback: give a small default set rather than everything
    return ["search_products", "get_price"]


def build_tools_for_request(user_message: str) -> list[dict]:
    relevant_names = select_relevant_tools(user_message)
    return [ALL_TOOLS[name] for name in relevant_names if name in ALL_TOOLS]

# Usage: only the relevant tools go into the API call, not all 6+.
# tools_for_this_call = build_tools_for_request("How much does the red mug cost?")
# -> only sends the get_price tool, so the model can't confuse it with search_products.

3. Parallel Tool Calls and Failure Handling

3.1 The simple example first

Suppose you ask a travel-planning friend: "Can you check the weather in Paris, Tokyo, and New York for me?" A slow friend would check Paris, wait, tell you, then check Tokyo, wait, tell you, then check New York. A smart friend instead opens three browser tabs at once, checks all three simultaneously, and gives you all three answers together. Same total work, far less waiting.

Now imagine one of those three lookups fails — say the New York weather site is down. Does your friend refuse to tell you about Paris and Tokyo just because New York failed? Or do they hand you what they do have, and mention New York didn't work? That's a judgment call — and it's the same judgment call your application has to make about tool calls.

3.2 Technical explanation

Parallel tool calls: the notes describe a model that can emit multiple tool-call requests in a single turn — e.g., "compare weather across N cities" or a "trip plan" that needs both search_flight and search_hotel at once. Your orchestration layer (the application code, not the model) is responsible for "fanning out" these calls concurrently, collecting all the results, and sending them back together. The direct payoff: it improves overall time to completion — you pay for the slowest of the N calls, not the sum of all N. The notes' guidance: use this whenever the operations are independent of one another (one call's result doesn't depend on another's).

Failure handling — "it depends". The notes are refreshingly honest here: "Answer to any question could well be IT DEPENDS." There isn't one universal right answer; it's a judgment call based on your product's tolerance for incompleteness vs. correctness. Three named strategies:

Prevent — stop the bad outcome before it happens. E.g., validate inputs up front, apply timeouts, use circuit breakers so a known-broken downstream tool isn't even attempted.
Absorb — let the failure happen, but contain the blast radius: send back only the successful tool-call responses and quietly drop/flag the failed ones, so the model can still produce a partial, useful answer rather than crashing the whole turn.
Fail gracefully — if absorbing isn't safe (e.g., you can't answer any meaningful part of the question without the missing piece), abort the entire execution and return a clear, honest error/refusal rather than a partial or misleading answer.

The notes give a concrete example of when parallel fan-out matters: "4 websearch calls, lookup across multiple knowledge bases." If a question needs facts from 4 different knowledge bases, firing all 4 lookups concurrently instead of one after another is the entire difference between a fast, responsive system and a sluggish one.

3.3 Code: concurrent tool calls with all three failure strategies

import asyncio
import random

# ---------------------------------------------------------------------------
# Mock tool: fetch weather for one city. Sometimes fails, to demonstrate
# failure handling (simulating a flaky downstream API).
# ---------------------------------------------------------------------------
async def fetch_weather(city: str) -> dict:
    await asyncio.sleep(random.uniform(0.1, 0.5))  # simulate network latency
    if city == "New York" and random.random() < 0.5:
        raise ConnectionError(f"Weather API timed out for {city}")
    return {"city": city, "temp_c": random.randint(10, 35)}


# =============================================================================
# STRATEGY 1: PREVENT
# Validate/guard BEFORE attempting the call, so we never even try a call
# we already know is unsafe (e.g. an unsupported city, a rate-limited tool).
# =============================================================================
SUPPORTED_CITIES = {"Paris", "Tokyo", "New York", "London"}

async def call_with_prevention(city: str) -> dict:
    if city not in SUPPORTED_CITIES:
        # We "prevent" the failure by never calling the API for a city we
        # know isn't supported, instead of letting it fail downstream.
        return {"city": city, "error": "unsupported_city", "skipped_call": True}
    return await fetch_weather(city)


# =============================================================================
# STRATEGY 2: ABSORB
# Run all calls concurrently; if one fails, catch the exception, keep going,
# and return only the successful results (plus a note about what failed).
# =============================================================================
async def call_with_absorption(cities: list[str]) -> dict:
    tasks = [fetch_weather(city) for city in cities]
    # return_exceptions=True means a failed task returns its Exception object
    # instead of crashing the whole asyncio.gather() call.
    results = await asyncio.gather(*tasks, return_exceptions=True)

    successes, failures = [], []
    for city, result in zip(cities, results):
        if isinstance(result, Exception):
            failures.append({"city": city, "error": str(result)})
        else:
            successes.append(result)

    return {"successful_results": successes, "failed_cities": failures}


# =============================================================================
# STRATEGY 3: FAIL GRACEFULLY
# If ANY call fails and a partial answer would be misleading (e.g. this tool
# call was mandatory context for the user's question), abort entirely and
# surface a clear, honest message instead of a silently incomplete answer.
# =============================================================================
async def call_with_graceful_failure(cities: list[str]) -> dict:
    tasks = [fetch_weather(city) for city in cities]
    try:
        results = await asyncio.gather(*tasks)  # no return_exceptions: first failure raises
        return {"ok": True, "results": results}
    except Exception as exc:
        return {
            "ok": False,
            "refused": True,
            "reason": f"Could not complete weather comparison: {exc}",
        }


async def main():
    cities = ["Paris", "Tokyo", "New York"]

    print("PREVENT strategy (one unsupported city):")
    print(await asyncio.gather(*[call_with_prevention(c) for c in ["Paris", "Atlantis"]]))

    print("\nABSORB strategy (best-effort, partial results ok):")
    print(await call_with_absorption(cities))

    print("\nFAIL GRACEFULLY strategy (all-or-nothing):")
    print(await call_with_graceful_failure(cities))


if __name__ == "__main__":
    asyncio.run(main())

Which strategy to pick is a product decision, not a technical one — that's the "it depends" in the notes. A weather-comparison feature can tolerate "absorb" (2 out of 3 cities is still useful). A financial transaction pipeline probably needs "fail gracefully" (a half-completed transfer is dangerous, not just incomplete).

4. Hybrid Search (Dense + Sparse Retrieval)

4.1 The simple example first

Say you're searching your email for something. If you search "birthday party invite," a smart search that understands meaning (semantic search) will find an email titled "Come celebrate turning 30!" even though none of your exact words appear in it — because it understands the concept of a birthday party.

But now say you search for an exact error code, like 1099-MISC. A meaning-based search might get confused and return generic tax documents about "how do I reset my password" style topics — it doesn't realize you need an exact string match, not a conceptual one. A classic old-school keyword search (like Ctrl+F, but smarter) would nail this instantly because it's built for exact-term matching.

Neither approach alone is good at everything. The fix: run both kinds of search at the same time, and combine (merge and rerank) their results — a bit like asking two friends with different strengths for restaurant recommendations, and combining their two rankings into one better, more trustworthy ranking.

4.2 Technical explanation

Dense retrieval (semantic search): your query and your documents are both converted into vectors ("embeddings") that capture meaning. Two pieces of text that mean similar things end up as vectors that are close together in space, even if they don't share any exact words. This is great when there's genuine semantic similarity — the notes' example: "How do I reset my password?" naturally matches a document titled "Account Recovery Steps," even with zero overlapping words.

Sparse retrieval / BM25: this is the notes' name for traditional keyword-based search ("BM25" = "Best Matching 25," a well-known statistical formula scoring how well a document's exact terms match a query's exact terms, weighted by term rarity and frequency). BM25 is the opposite strength of dense search: it's excellent at exact terms, codes, IDs, acronyms — things like "error code 1099-MISC" — but it has no concept of meaning, so it's poor at "How do I reset my password?" style natural-language questions where the right document doesn't share your exact wording.

Hybrid search = dense + sparse, run together. The notes' rule of thumb: "20|50 documents certainly having the right answer" — hybrid retrieval improves recall (the fraction of genuinely relevant documents that get retrieved at all), because you're covering both failure modes (semantic gap AND exact-term gap) simultaneously. The fix line from the notes: "Run both and merge and rerank!"

Reciprocal Rank Fusion (RRF) is the merging technique. Analogy: imagine two friends each independently rank the same 10 restaurants from 1 (best) to 10 (worst). To get one combined ranking that reflects both opinions, you don't just average their scores (their internal scoring "scales" may be totally different: one friend rates out of 5 stars, another out of 100 points). Instead, RRF says: only the rank position matters, not the internal score — a restaurant ranked #1 by both friends should shoot to the top of the combined list, even if their raw scores were on incompatible scales. This "brings reciprocal ranks together" as the notes put it.

The formula, straight from the notes:

RRF(d) = Σ  1 / (k + rank_r(d))
         r∈R

Read this piece by piece:

d is a particular document you're scoring.
R is the set of ranked result lists you're fusing (in our case: the dense-search ranking and the BM25 ranking).
rank_r(d) is the position of document d in ranking r (1st place, 2nd place, etc.).
k is a small constant, usually 60 (per the notes) — it dampens the effect of very high ranks so a #1 spot doesn't dominate too extremely, and it also means documents that don't appear at all in one of the lists aren't catastrophically penalized.
You take 1 / (k + rank) for each list the document appears in, and sum those fractions across all lists. A document ranked highly (small rank number) in multiple lists gets a bigger combined score.

The notes' visual intuition: dense list gives fractions like 1/1, 1/2, ... and sparse list gives 1/61, 1/62, ... (i.e., 1/(60+1), 1/(60+2), ...) — "Fusion is better" because a document doesn't need to win outright in either individual list; it just needs to consistently rank reasonably well across both to accumulate a high combined score.

One more practical note from the notes: Weaviate and Elasticsearch (ES) support hybrid search natively — meaning they've built RRF-style fusion directly into the database. Otherwise, you have to do it yourself with a scatter-gather pattern (fire off both queries yourself, gather both result sets back in your application, then fuse them).

4.3 Code: RRF fusion from scratch (no external libraries)

# =============================================================================
# Reciprocal Rank Fusion (RRF) — implemented from the raw formula.
# RRF(d) = sum over each ranked list r containing d of  1 / (k + rank_r(d))
# =============================================================================

def reciprocal_rank_fusion(ranked_lists: list[list[str]], k: int = 60) -> list[tuple[str, float]]:
    """
    ranked_lists: a list of ranked result lists. Each inner list is already
                  sorted best-to-worst, e.g. ["doc3", "doc1", "doc9"].
    k: the RRF damping constant (60 is the standard default from the notes).

    Returns a list of (doc_id, fused_score) sorted from best to worst.
    """
    scores: dict[str, float] = {}

    for ranked_list in ranked_lists:
        for position, doc_id in enumerate(ranked_list):
            rank = position + 1  # ranks start at 1, not 0
            # This is the exact formula: 1 / (k + rank)
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)

    # Sort documents by their fused score, highest first
    fused_ranking = sorted(scores.items(), key=lambda pair: pair[1], reverse=True)
    return fused_ranking


# --- Example usage ---------------------------------------------------------

# Dense (semantic) search results for "how do I reset my password?"
dense_results = ["account_recovery_steps", "login_faq", "security_settings", "billing_faq"]

# BM25 (keyword) search results for the SAME query — note it ranks things
# differently because it only sees exact word overlap
sparse_results = ["login_faq", "account_recovery_steps", "terms_of_service", "billing_faq"]

fused = reciprocal_rank_fusion([dense_results, sparse_results], k=60)

for doc_id, score in fused:
    print(f"{doc_id}: {score:.5f}")

# Expected behaviour: "account_recovery_steps" and "login_faq" both rank
# highly in BOTH lists, so their fused scores rise above documents that only
# appeared in one list or ranked poorly in both.

4.4 Code: a BM25 call + a vector similarity call, combined

# This shows the SHAPE of a real hybrid search call: one sparse (BM25) query,
# one dense (vector) query, run independently, then fused with RRF.
# (rank_bm25 and sentence-transformers are common lightweight libraries.)

# pip install rank_bm25 sentence-transformers --break-system-packages
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
import numpy as np

documents = [
    "To reset your password, go to Account Recovery and click 'Forgot Password'.",
    "Error code 1099-MISC indicates a mismatch in the tax reporting form.",
    "Our billing FAQ covers refunds, invoices, and subscription changes.",
]
doc_ids = ["account_recovery", "error_1099_misc", "billing_faq"]

query = "how do I reset my password"

# --- Sparse (BM25) leg: pure keyword/token overlap scoring -----------------
tokenized_corpus = [doc.lower().split() for doc in documents]
bm25 = BM25Okapi(tokenized_corpus)
bm25_scores = bm25.get_scores(query.lower().split())
sparse_ranking = [doc_ids[i] for i in np.argsort(bm25_scores)[::-1]]

# --- Dense (semantic) leg: embed query + docs, rank by cosine similarity ---
embedder = SentenceTransformer("all-MiniLM-L6-v2")  # small, fast embedding model
doc_embeddings = embedder.encode(documents, normalize_embeddings=True)
query_embedding = embedder.encode(query, normalize_embeddings=True)
cosine_scores = doc_embeddings @ query_embedding  # dot product of normalized vectors = cosine similarity
dense_ranking = [doc_ids[i] for i in np.argsort(cosine_scores)[::-1]]

print("Sparse (BM25) ranking:", sparse_ranking)
print("Dense (semantic) ranking:", dense_ranking)

# --- Fuse both rankings with RRF (reusing the function from section 4.3) ---
final_ranking = reciprocal_rank_fusion([dense_ranking, sparse_ranking], k=60)
print("Final hybrid ranking:", final_ranking)

5. Metadata Filtering

5.1 The simple example first

Imagine you're searching your email for a specific message and you know it arrived last week. You could (a) tell your email client "only show me emails from last week," and then search the text — that's fast and narrow. Or you could (b) search the text across your entire mailbox history, get back thousands of results, and manually throw away everything not from last week — that's slow and wasteful, since your search engine did a ton of unnecessary work scanning old emails it was always going to discard.

Now a second, more serious example: at a company, an HR employee should be able to see every department's HR documents, but a regular software engineer should only see documents relevant to their own team — they should never even see, let alone search inside, another department's confidential HR files. This isn't just a nice-to-have; it's a hard security requirement, often called RBAC (Role-Based Access Control — a plain-English one-liner: "restricting what a user can see or do based on their assigned role").

5.2 Technical explanation

Metadata filtering applies hard constraints (like date ranges, document owner, department, tags) either before or after the vector search runs, on top of whatever relevance ranking your retrieval produces. The notes' example query: "What was our Q1 2026 revenue?" — you'd want to filter to date range = Q1 2026 before even letting semantic search loose on the whole corpus.

Beyond just retrieval quality, the notes stress this is especially important for Enterprise RAG and RBAC: an employee simply must not be able to retrieve another department's HR docs, no matter how semantically relevant those docs might look to their query. This is a security boundary, not just a relevance tweak.

There are two places you can apply the filter:

Pre-filter — you extract the filter conditions (date range, department, tags) and pass them into the vector search itself, so the search engine only ever looks at the allowed subset of documents. The notes call this faster, but note it requires that "the vector db should support metadata filter" natively (not every vector database can efficiently combine filtering with approximate nearest-neighbor search).
Post-filter — you run the vector search across the entire corpus first, get back your top-K results, and then throw away any results that don't match your filter. The notes call this a waste of retrieval budget: you spent compute finding results you were always going to discard, and worse, you might discard so many that you end up with fewer than K valid results left over (imagine 8 of your top 10 semantic matches belong to a forbidden department — post-filtering leaves you with only 2 usable results).

5.3 Code: pre-filter (vector DB metadata filter) vs. post-filter (Python)

# =============================================================================
# PRE-FILTERING: the filter is passed INTO the vector database query itself.
# The DB never even considers documents outside the RBAC/date boundary, so
# ranking + filtering happen together, efficiently, in one pass.
# (Qdrant-style filter syntax, as referenced in the notes' prototype.)
# =============================================================================

# pip install qdrant-client --break-system-packages
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue, Range

client = QdrantClient(url="http://localhost:6333")  # assumes a running Qdrant instance

query_vector = [0.12, 0.98, 0.31]  # a real embedding would be much longer, e.g. 1536-dim

results = client.search(
    collection_name="hr_documents",
    query_vector=query_vector,
    query_filter=Filter(
        must=[
            # RBAC constraint: only documents this user's department may see
            FieldCondition(key="department", match=MatchValue(value="engineering")),
            # Date constraint: only Q1 2026 documents
            FieldCondition(key="date", range=Range(gte="2026-01-01", lte="2026-03-31")),
        ]
    ),
    limit=10,
)
# Qdrant applies the filter WHILE searching, so all 10 results returned are
# guaranteed to satisfy both constraints — no wasted compute, no leakage risk.


# =============================================================================
# POST-FILTERING: search everything first, THEN discard invalid results in
# plain Python. Shown for contrast — this is what the notes call "wasteful."
# =============================================================================

def post_filter_search(all_search_results: list[dict], user_department: str) -> list[dict]:
    """
    all_search_results: e.g. the top-K matches from an UNFILTERED vector search
                         across the entire corpus, regardless of department.
    """
    filtered = [
        doc for doc in all_search_results
        if doc["department"] == user_department
        and "2026-01-01" <= doc["date"] <= "2026-03-31"
    ]
    return filtered

# Danger illustrated: if the unfiltered top-10 search returns mostly documents
# from OTHER departments (because they happened to be semantically similar),
# post-filtering might leave you with just 1-2 usable results, or worse —
# if the filtering logic has a bug, a forbidden document could slip through
# into the LLM's context before being filtered, which is a real security risk
# with RBAC-sensitive data. Pre-filtering avoids this entirely.

6. Reranking: Cross-Encoders vs. Bi-Encoders

6.1 The simple example first

Bi-encoder analogy: imagine two strangers each write their own short bio independently — "I like hiking, sci-fi movies, and cooking Italian food" and "I enjoy the outdoors, watching space films, and making pasta." You then compare these two bios after the fact to guess how compatible these two people might be. It's fast (you can pre-write and store millions of bios ahead of time and just compare later), but it's a bit shallow — the two people never actually talked to each other, so subtle nuances get missed.

Cross-encoder analogy: now imagine instead you sit those same two strangers down together for an actual conversation. They can react to each other in real time, clarify ambiguous statements, and pick up on nuance that never would have shown up in two separately-written bios. You get a much better read on true compatibility — but it only works one pair at a time, and a live conversation takes real time and effort per pair. You couldn't have every person in a stadium have a real conversation with every other person — it simply wouldn't scale.

6.2 Technical explanation

Bi-encoders (used for the first stage of dense retrieval, described back in Section 4): the query is turned into an embedding, and each document is turned into an embedding, completely independently of each other. Relevance is then just vector proximity — how close the two embeddings are (measured via cosine similarity). Because documents can be embedded once, offline, ahead of time, bi-encoders are fast at query time and scalable — this is why they're "good enough for most" use cases, per the notes.

Their key weakness, from the notes: "No cross attention, i.e. query and doc never see each other." Because the two embeddings were produced in isolation, the model can't attend to specific interactions between query terms and document terms. This makes bi-encoders weaker on negation, exact terms, etc. — e.g., a bi-encoder might not clearly distinguish "the flight is NOT delayed" from "the flight IS delayed," because both sentences produce very similar embeddings overall (they share almost all the same words).

Cross-encoders, by contrast, take the <query, document> pair as a single input — literally concatenating the query and the candidate document together — and output one single relevance score for that specific pair. Because "every token in the query attends every token of the document" (full cross-attention, across the whole concatenated input), the model can compare query and document together, catching subtleties bi-encoders miss. This: improves precision, and it genuinely sees the query and document together rather than as two isolated summaries.

The notes draw a small attention diagram illustrating the "lost in the middle" problem: attention/relevance tends to concentrate most heavily near the query token and decays as you move deeper into a long document — meaning information buried in the middle of a long document or a long context window can get comparatively under-weighted relative to information at the start or end. This is a known failure mode of transformer attention over long inputs, and it's part of why reranking with a small, focused candidate set (rather than dumping a giant unranked context at the LLM) genuinely helps.

Cross-encoders: much higher accuracy — they can compare every token in the query to every token in the document, so they handle negation, exact matches, etc. far better than bi-encoders. Example model named in the notes: cross-encoder/ms-marco-MiniLM-L-6-v2.

The catch — cross-encoders are slow and don't scale. From the notes: they work well on "50 doc[s] v[s] million doc[s] per query" — meaning cross-encoders are only practical on a small shortlist, never the entire corpus. There's no indexing possible (because the score depends on the specific query, you can't precompute it ahead of time the way you can with bi-encoder embeddings), and each pair costs roughly ~10ms of latency.

The latency math, worked out step by step (from the notes):

Assume you have 1 million documents and it costs 20ms per <query, doc> pair to score with a cross-encoder.
Total time for one query = 1,000,000 docs × 20ms = 20,000,000ms = 20,000 seconds.
Converting to hours: 20,000 seconds ÷ 3,600 seconds/hour ≈ 5.5 hours per single query.

That's obviously unusable in a real product. The fix, stated directly in the notes: "could be done on candidate docs only." This is exactly why the two techniques are used together in a pipeline, not as alternatives: use the cheap, scalable bi-encoder (plus BM25/hybrid search) to narrow millions of documents down to a small shortlist (say, the top 40), and then run the expensive, high-accuracy cross-encoder reranker only on that small shortlist. You get bi-encoder speed at scale, and cross-encoder accuracy where it matters most — the final ranking of the few documents that actually make it into the LLM's context.

6.3 Code: bi-encoder vs. cross-encoder on the same toy dataset

# pip install sentence-transformers --break-system-packages
from sentence_transformers import SentenceTransformer, CrossEncoder
import numpy as np

query = "Why is my API returning a 500 error under high load?"

candidate_docs = [
    "Throughput degradation under load can trigger unhandled exceptions, "
    "which surface to callers as HTTP 500 errors.",
    "To reset your password, click 'Forgot Password' on the login screen.",
    "500 errors are NOT related to client-side input validation issues.",
    "Our billing FAQ covers refunds, invoices, and subscription cancellations.",
]

# =============================================================================
# STAGE 1 (fast, scalable): BI-ENCODER — embed query and docs SEPARATELY,
# then compare with cosine similarity. This is what you'd run across
# millions of documents to get an initial shortlist.
# =============================================================================
bi_encoder = SentenceTransformer("all-MiniLM-L6-v2")

query_vec = bi_encoder.encode(query, normalize_embeddings=True)
doc_vecs = bi_encoder.encode(candidate_docs, normalize_embeddings=True)

# cosine similarity = dot product of normalized vectors
bi_encoder_scores = doc_vecs @ query_vec

print("Bi-encoder ranking (fast, first-pass):")
for idx in np.argsort(bi_encoder_scores)[::-1]:
    print(f"  {bi_encoder_scores[idx]:.3f}  {candidate_docs[idx][:60]}...")

# =============================================================================
# STAGE 2 (slow, accurate): CROSS-ENCODER — feed the query+doc TOGETHER as a
# single pair, get one relevance score per pair. Only run this on the small
# shortlist that stage 1 already narrowed down (here, all 4, since it's tiny).
# =============================================================================
cross_encoder = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

pairs = [[query, doc] for doc in candidate_docs]  # literal <query, doc> pairing
cross_encoder_scores = cross_encoder.predict(pairs)

print("\nCross-encoder ranking (slow, high-precision rerank):")
for idx in np.argsort(cross_encoder_scores)[::-1]:
    print(f"  {cross_encoder_scores[idx]:.3f}  {candidate_docs[idx][:60]}...")

# You'll typically notice the cross-encoder more confidently separates the
# TRUE match (the throughput/500-error doc) from the negation trap (doc #3,
# which mentions "500 errors" but explicitly says they are NOT related to
# the topic) — a distinction bi-encoders are known to struggle with.

7. Query Rewriting with HyDE

7.1 The simple example first

Imagine you ask a librarian, "Why is my code slow?" The librarian's catalog, though, doesn't use casual words like "slow" — the technical manuals on the shelf are indexed under formal terms like "throughput degradation," "latency regression," or "bottleneck analysis." If the librarian searches the catalog using your exact casual phrase, they might get nothing useful back, even though the perfect manual for your problem exists on the shelf. The words just don't line up.

A clever librarian's trick: before searching, they first jot down (in their head) roughly what the ideal, technical-sounding manual would probably say about your problem — then they search the catalog using that imagined, more formal description instead of your casual phrasing. That imagined "ideal answer" acts as a bridge between your everyday words and the library's formal vocabulary.

That's exactly what HyDE (Hypothetical Document Embedding) does for a RAG system.

7.2 Technical explanation

The notes call the "Query-Document Gap" the most persistent failure mode in RAG. It shows up because user queries tend to be short and conversational, while the underlying documents/corpus are often semantically distant in their vocabulary — the classic example: "why is my pipeline slow?" (user's words) vs. "Throughput degradation is..." (the document's actual wording). Even though these mean almost the same thing conceptually, their raw embeddings may not be close enough for a naive semantic search to reliably connect them.

There are two named fixes in the notes:

1. Expand or rephrase the user query. Simply ask an LLM: "generate 3 alternate phrasings..." of the user's question, and search using all of them (or a combination), increasing the odds that at least one phrasing's embedding lands close to the real document.

2. HyDE — Hypothetical Document Embedding. Instead of embedding the user's raw, short question, you first ask an LLM to write a short document that would answer this question — the exact prompt style from the notes: "Write a short document that will answer this question. Be factual. If you don't know the answer, write what the answer would typically look like." You then take that generated hypothetical document — not the original question — and embed it. Because this hypothetical answer is written in a style and vocabulary much closer to how real documents in your corpus are likely written, its embedding tends to land much closer to the real relevant document's embedding than the original short, casual question would have.

Technically, HyDE reframes retrieval as a similarity search between d and d ∈ D — that is, comparing a (hypothetical) document to the real documents in your corpus D, rather than comparing a short question to documents (which is a fundamentally harder, more mismatched comparison).

When does this actually help? The notes are precise about this: HyDE "works when corpus vocab is very different than user query." Good candidates: scientific papers, legal texts, internal jargon-heavy documentation — domains where the "real" documents are written in dense, formal, or highly technical language that a typical user's question would never naturally use.

7.3 Code: HyDE retrieval function

# pip install anthropic sentence-transformers --break-system-packages
import anthropic
from sentence_transformers import SentenceTransformer
import numpy as np

client = anthropic.Anthropic()
embedder = SentenceTransformer("all-MiniLM-L6-v2")


def generate_hypothetical_answer(user_query: str) -> str:
    """
    Ask an LLM to write a short, FACTUAL-SOUNDING document that would answer
    the user's question. This is the exact prompt shape from the notes:
    "Write a short document that will answer this question. Be factual.
    If you don't know the answer, write what the answer would typically
    look like."
    We use this generated text purely to bridge the vocabulary gap — we do
    NOT show this hypothetical text to the end user; it's an internal
    retrieval-only artifact.
    """
    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=300,
        messages=[
            {
                "role": "user",
                "content": (
                    "Write a short document that will answer this question. "
                    "Be factual. If you don't know the answer, write what the "
                    f"answer would typically look like.\n\nQuestion: {user_query}"
                ),
            }
        ],
    )
    return response.content[0].text


def hyde_retrieve(user_query: str, doc_texts: list[str], doc_ids: list[str], top_k: int = 3):
    """
    Full HyDE retrieval: generate a hypothetical answer, embed THAT instead
    of the raw query, then find the closest real documents to it.
    """
    # Step 1: generate the hypothetical, jargon-matched answer
    hypothetical_doc = generate_hypothetical_answer(user_query)

    # Step 2: embed the HYPOTHETICAL document, not the raw user query
    hyde_embedding = embedder.encode(hypothetical_doc, normalize_embeddings=True)

    # Step 3: embed the real corpus documents (in production this is pre-computed
    # and stored in a vector DB, not re-computed on every query)
    doc_embeddings = embedder.encode(doc_texts, normalize_embeddings=True)

    # Step 4: rank real documents by similarity to the hypothetical embedding
    similarities = doc_embeddings @ hyde_embedding
    top_indices = np.argsort(similarities)[::-1][:top_k]

    return [(doc_ids[i], doc_texts[i], float(similarities[i])) for i in top_indices]


# --- Example -----------------------------------------------------------------
corpus_texts = [
    "Throughput degradation under sustained load is typically caused by "
    "connection pool exhaustion or unindexed database queries.",
    "Our refund policy allows cancellations within 30 days of purchase.",
    "Latency regressions in distributed systems often trace back to N+1 "
    "query patterns or missing caching layers.",
]
corpus_ids = ["perf_doc_1", "refund_policy", "perf_doc_2"]

results = hyde_retrieve("why is my code slow?", corpus_texts, corpus_ids)
for doc_id, text, score in results:
    print(f"{score:.3f}  {doc_id}: {text[:70]}...")

# Without HyDE, embedding the raw casual query "why is my code slow?" might
# rank the perf docs only weakly, since they never use the word "slow."
# With HyDE, the generated hypothetical answer naturally uses words like
# "throughput," "latency," and "degradation" -- much closer to the real docs.

8. Semantic Caching

8.1 The simple example first

Imagine ten different customers each phrase the exact same underlying question in slightly different words: "How do I integrate libX?", "I want help integrating libX," "How can I get libX working with my app?" A naive cache (the kind that only matches exact strings) would treat these as three totally unrelated questions and pay to compute a fresh, expensive answer for every single one — even though the right answer is identical for all three.

A smarter cache recognizes that all three questions mean the same thing, and serves the same cached, already-computed answer to all of them — saving time and money on the second and third askers, without them ever noticing anything different.

8.2 Technical explanation

The core motivating fact from the notes: "Every call to LLM cost[s] latency and tokens" — i.e., time and money. A semantic cache sits in front of the LLM: it intercepts a query before it reaches the model and, if a semantically similar (not necessarily an exact-text-match) prior query exists in the cache, it returns the cached response instead of paying for a new LLM call.

When this works best, per the notes: static data with no personalization — situations where the "right" answer genuinely doesn't change per-user or over short timeframes. Two named examples:

Asking an AI assistant about your API/integration documentation ("How do I integrate libX?" / "I want help to integrate libX" — both map to the same content already published on libX's website).
A customer-support chatbot with a bounded query space — the notes cite that in such settings, roughly 60% of incoming queries can be near-duplicates of previously-asked ones, making caching extremely high-leverage.

Implementation, step by step (from the notes):

Take the incoming query, and either normalize it (lowercase, strip punctuation, etc.) or store it as-is.
Index it in a vector database along with its (eventual) response — so both the question's embedding and its answer live together.
During lookup, find similar queries — embed the new incoming query, and search the cache's vector index for anything above a chosen similarity threshold.

The core tension: cost vs. correctness, governed entirely by what similarity threshold you pick and how you set it. The notes are blunt about the two failure directions:

Threshold too low → wrong answers. You'll match queries that only look superficially similar but actually mean something different (e.g., "How do I cancel my subscription?" vs. "How do I change my subscription plan?" — related, but the correct action is different), and you'll confidently serve the wrong cached answer.
Threshold too high → hit rate collapses. You become so strict that almost nothing ever matches, and you rarely benefit from caching at all, defeating the entire purpose.

The reliable way to pick a threshold, per the notes: don't guess — empirically find your "sweet spot" on your own query distribution. As a reasonable starting point, the notes suggest 0.85 cosine similarity, but this must be calibrated, not treated as gospel, because every product's query patterns differ.

8.3 Code: a minimal semantic cache class

# pip install sentence-transformers --break-system-packages
from sentence_transformers import SentenceTransformer
import numpy as np


class SemanticCache:
    """
    A minimal in-memory semantic cache: get() looks for a semantically
    similar prior query above `similarity_threshold`; set() stores a new
    query + response pair for future lookups.
    """

    def __init__(self, similarity_threshold: float = 0.85):
        self.embedder = SentenceTransformer("all-MiniLM-L6-v2")
        self.similarity_threshold = similarity_threshold
        self.cached_queries: list[str] = []
        self.cached_embeddings: list[np.ndarray] = []
        self.cached_responses: list[str] = []

    def get(self, query: str) -> str | None:
        """Return a cached response if a semantically similar query exists, else None."""
        if not self.cached_queries:
            return None

        query_embedding = self.embedder.encode(query, normalize_embeddings=True)
        similarities = np.array(self.cached_embeddings) @ query_embedding
        best_idx = int(np.argmax(similarities))
        best_score = float(similarities[best_idx])

        if best_score >= self.similarity_threshold:
            print(f"[cache HIT] similarity={best_score:.3f} matched: "
                  f"'{self.cached_queries[best_idx]}'")
            return self.cached_responses[best_idx]

        print(f"[cache MISS] best similarity was only {best_score:.3f}")
        return None

    def set(self, query: str, response: str) -> None:
        """Store a new query + response pair in the cache."""
        embedding = self.embedder.encode(query, normalize_embeddings=True)
        self.cached_queries.append(query)
        self.cached_embeddings.append(embedding)
        self.cached_responses.append(response)


# --- Demonstration: threshold too low vs. too high --------------------------

cache = SemanticCache(similarity_threshold=0.85)
cache.set("How do I integrate libX?", "To integrate libX, install it via pip and call libx.init()...")

print("\n--- Reasonable threshold (0.85) ---")
cache.get("I want help integrating libX")   # should HIT: same intent, different phrasing
cache.get("What's the weather in Tokyo?")   # should MISS: unrelated question

print("\n--- Threshold set TOO LOW (0.50): risks wrong answers ---")
loose_cache = SemanticCache(similarity_threshold=0.50)
loose_cache.set("How do I cancel my subscription?", "Go to Settings > Billing > Cancel Plan.")
# This is a RELATED but MEANINGFULLY DIFFERENT question -- with a low
# threshold it may incorrectly match and serve the wrong cached answer.
loose_cache.get("How do I upgrade my subscription plan?")

print("\n--- Threshold set TOO HIGH (0.99): hit rate collapses ---")
strict_cache = SemanticCache(similarity_threshold=0.99)
strict_cache.set("How do I integrate libX?", "To integrate libX, install it via pip...")
# Even a near-identical paraphrase may now fail to match because embeddings
# are almost never PERFECTLY identical between two different sentences.
strict_cache.get("I want help to integrate libX")

The prototype in the notes ("Semantic Caching and Similarity Calibration") describes exactly this experiment at scale: 500 queries → embeddings → index, then sweeping thresholds from 0.75 to 0.95, populating the cache first, then computing hit rate (fraction of queries found in cache) and accuracy/precision. It uses three types of test queries: seed (the original query, e.g. "Explain the ending of movie Arrival"), paraphrase (same intent, different words, e.g. "Please clarify the conclusion of the movie Arrival" — a correct match), and near-miss (looks similar but has a genuinely different intent, e.g. "Who directed the movie Arrival" — a trap, since matching this to the seed's answer would be a false positive / incorrect hit). The accuracy formula from the notes:

accuracy = correct hits (paraphrase matching the seed) / total hits

Below is a small script that runs this exact style of calibration experiment:

def evaluate_threshold(cache: SemanticCache, seed_paraphrase_pairs, near_miss_queries):
    """
    seed_paraphrase_pairs: list of (seed_query, paraphrase_query, response) tuples
                           -- paraphrase SHOULD correctly hit the seed's cached entry.
    near_miss_queries: list of queries that LOOK similar but have different intent
                       -- these SHOULD ideally miss (or if they hit, that's a false positive).
    """
    for seed, response in {(s, r) for s, _, r in seed_paraphrase_pairs}:
        cache.set(seed, response)

    total_hits, correct_hits = 0, 0

    for seed, paraphrase, _ in seed_paraphrase_pairs:
        result = cache.get(paraphrase)
        if result is not None:
            total_hits += 1
            correct_hits += 1  # paraphrase hitting its own seed = correct

    false_positives = 0
    for near_miss in near_miss_queries:
        result = cache.get(near_miss)
        if result is not None:
            total_hits += 1
            false_positives += 1  # near-miss hitting ANY cached entry = wrong

    accuracy = correct_hits / total_hits if total_hits else 0
    return {"hit_rate_relevant": correct_hits, "false_positives": false_positives, "accuracy": accuracy}

9. Full System Design: RAG with 10M Documents, Zero Hallucinations

9.1 The simple example first

Imagine building a "help desk" for a huge company with 10 million internal documents. A visitor walks up and asks a question in plain English. Before answering, your help desk clerk must: (1) understand what's really being asked, (2) go search both the exact-keyword index and the meaning-based index of every document, (3) combine and re-sort those results by true relevance, (4) double-check that the results are actually confident and trustworthy (not just "vaguely related"), (5) draft an answer strictly grounded in what was found, and — critically — (6) go back and check every single sentence of that draft answer against the source documents before saying it out loud, refusing to say anything it can't verify. That's the whole system in one sentence: retrieve carefully, answer carefully, and never say something you can't prove.

9.2 Technical explanation: requirements and why they matter

What the system does: Answers natural language queries. The one hard guarantee that shapes the entire design: "the system must not assert any claim that cannot be traced to [a] retrieved passage." This single sentence is why "correctness" (not just relevance, not just speed) drives nearly every architectural decision below. Why this guarantee matters: an internal knowledge system that confidently states wrong facts is often worse than one that admits "I don't know," because wrong confident answers erode trust and can cause real harm (wrong policy info, wrong financial numbers, etc.).

What the system explicitly is NOT: a document management system, an ingestion pipeline product, etc. — the notes flag this so the design stays scoped: we are not building a general-purpose CMS, just the "ask questions about documents that already exist somewhere" layer.

Scale numbers (why they matter):

10 million documents, roughly 2KB each → this sets the raw data size baseline (worked out below).
200 QPS (queries per second) → this is very read-heavy traffic, which shapes almost every capacity and replication decision (lots of replicas for reads, less concern about write throughput).
~1,000 documents updated per day → writes are comparatively rare, which is why the system can tolerate eventual consistency rather than needing instant read-your-writes guarantees.
We do not need to serve data from a just-updated document immediately — an SLO (service level objective, a plain-English promise about how good the system must be) of roughly ~15 minutes eventual consistency is acceptable. Why this matters: it means the write path doesn't need to be synchronously blocking or ultra-low-latency; it can batch, chunk, and index asynchronously.

9.3 The nine solutioning questions (from the notes)

The notes lay out the exact ordered checklist a systems-design interview or real design doc would walk through:

How do we store the docs, and where?
What does the query API look like?
What does the response look like?
Which databases, and why?
Capacity estimation: query and storage.
Read path and query pseudocode.
What should be returned?
What are the failure modes in the read path?
What does the write path look like (S3 → databases)?
What are we missing? (a deliberate closing gut-check)

Let's walk through each, grounding every architectural choice in a "why" before the "what."

1 & 2 — Storage location and Query API. Documents live in an S3 bucket (e.g. s3://myrag/doc1.txt), organized per the notes with some inspiration from Git's object storage model (content-addressed, organized by structure like date/git-style hashing prefixes, e.g. folders ab/, ac/, ad/ — a common trick to avoid dumping literally 10 million flat files into one folder, which many filesystems and object stores handle poorly). The query API is a simple HTTP endpoint: GET /query? — the notes sketch this as @app.get('/query?').

3 — Response shape. From the notes, a single respond() call returns a structured object containing:

question: <string>
filters: { date_from, date_to, source_domains, tag, topics, etc. }
answer: <string>
refused: <true/false>
refusal_reason: <string, if refused>
passages: [ {doc_id, passage_id, text, score, highlight_idx}, ... ]
confidence: <number>
query_id: <string>   -- used for tracing/debugging a specific request later

Why this shape: it's not enough to return just an answer string. You need the passages (so a user or auditor can verify the claim was actually grounded), a refused boolean plus reason (so "I don't know" is a first-class, explicit outcome, not a vague hedge buried in the answer text), a confidence score (so downstream systems or UI can decide whether to trust/display the answer), and a query_id for tracing — being able to look up exactly what happened for a specific request later, which is essential for debugging a "why did it hallucinate here" incident.

4 — Which databases, and why:

S3 — the durable source of truth for raw documents.
Postgres (PG Master + Replica) — stores document and passage metadata (id, text, title, URL, date, tags, is-deleted flag for docs; id, doc_id, chunk_index, text, char_start, char_end, token_count for passages). Why Postgres: this is classic structured, relational metadata — dates, tags, foreign keys between docs and their passages — exactly what a relational database is built for, plus JSONB support for flexible tag/topic fields and strong replication for read-heavy traffic.
Qdrant (vector database) — stores the dense embeddings for approximate nearest-neighbor (ANN) search. Why a dedicated vector DB: embeddings need specialized indexing structures (like HNSW — Hierarchical Navigable Small World, a plain-English one-liner: "a graph-based index structure that lets you find approximately the closest vectors in high-dimensional space without comparing against every single vector") that general-purpose databases don't do efficiently at this scale.
ElasticSearch (ES) — holds the BM25 inverted index for the sparse/keyword leg of hybrid search (Section 4), using an English analyzer with stemming and stop-word removal (so "running" and "run" are treated as the same term, and words like "the"/"is" don't dilute matching).
Redis (cache) — caches passages and query results (this is the semantic-caching layer from Section 8), reducing repeat load on the expensive LLM-generation step.

5 — Capacity estimation, worked step by step (numbers straight from the notes' math):

Query load:

50,000 DAU (daily active users) × 8 queries/day/user × 4.5 (peak multiplier)
──────────────────────────────────────────────────────────────────────────  = 200 QPS
                          86,400 seconds/day

So: 50,000 × 8 = 400,000 total queries/day. 400,000 ÷ 86,400 seconds ≈ 4.6 QPS average. Multiply by a peak multiplier of 4.5 (because traffic isn't spread evenly across 24 hours — there are busy hours) to get roughly 200 QPS at peak. The team then designs for a bit of headroom: 250 QPS target capacity.

Storage — raw documents:

10,000,000 docs × 2KB each = 20GB raw data

This is small enough to be almost trivial to store redundantly — it's the derived data (embeddings, indexes) that actually dominates storage, as shown next.

Storage — vector embeddings:

10 × 10^6 documents × 8 passages/doc (avg) × 1536 dimensions × 4 bytes/float32
= 10×10^6 × 8 × 1536 × 4 bytes
≈ 491 GB (just the raw embedding vectors)

Then add HNSW graph overhead (the extra data structure the ANN index needs to do fast approximate search) — roughly 1.5x — bringing:

Total vector index storage ≈ 491GB × 1.5 ≈ 700GB

This is why the notes conclude: "Hence, we need a distributed vector database" — 700GB (spread across replicas for redundancy and read throughput) is well beyond what a single machine comfortably handles for a low-latency, always-on service. The notes also mention Product Quantization can reduce this by roughly 8x, but "at [the] cost of recall" — i.e., you trade some retrieval accuracy for a much smaller memory footprint, a classic space/accuracy tradeoff.

ElasticSearch (BM25 index) sizing: roughly 20GB × 1.5 ≈ 30GB for the inverted index, run as a 3-node cluster with high availability (HA) plus 3 replicas, ≈ 90GB total, each node with 30GB RAM.

Postgres sizing: the 20GB of metadata plus JSONB overhead and replication comes out to roughly 60-80GB, served with 3 read replicas (since this is read-heavy traffic) with a target replication lag of under 100ms.

Redis: a 3-node cluster with 3 replicas, caching passages and query results — this is explicitly conditioned on the earlier semantic-caching insight: it works best "if no personalization & data is static."

6, 7, 8 — Read path, what's returned, and failure handling are covered together via the code walkthrough in Section 9.4 below (the notes' own respond() pseudocode).

9 — Write path is covered in Section 9.5 below.

9.4 Code: the read path (`respond()` function), cleaned up and fully commented

The notes' own hand-drawn pseudocode for the read path is the heart of the "zero hallucinations" guarantee. Here it is translated into clean, fully-commented Python:

from dataclasses import dataclass, field


@dataclass
class RespondResult:
    answer: str
    passages: list
    refused: bool
    refusal_reason: str | None = None
    confidence: float | None = None


def respond(question: str) -> RespondResult:
    """
    The full "zero hallucinations" read path:
    embed -> hybrid search (dense + sparse) -> RRF fusion -> cross-encoder
    rerank -> confidence threshold -> LLM generate -> claim extraction ->
    NLI fact-check -> final answer OR refusal.

    Every stage is a chance to refuse rather than guess -- that's the whole
    point of the "must not assert unsupported claims" guarantee.
    """

    # -------------------------------------------------------------------
    # Step 1: Embed the question (turn text into a dense vector).
    # -------------------------------------------------------------------
    query_vector = embed(question)

    # -------------------------------------------------------------------
    # Step 2: Hybrid search -- run dense (vector) and sparse (BM25) search
    # CONCURRENTLY (this is the "parallel tool calls" idea from Section 3
    # applied to retrieval itself), each returning their own top-40 shortlist.
    # -------------------------------------------------------------------
    dense_results = qdrant_ann_search(query_vector, top_k=40)
    sparse_results = elasticsearch_bm25(question, top_k=40)

    # -------------------------------------------------------------------
    # Step 3: Fuse both ranked lists with Reciprocal Rank Fusion (Section 4).
    # -------------------------------------------------------------------
    candidates = rrf_fusion(dense_results, sparse_results)

    # -------------------------------------------------------------------
    # Step 4: Rerank the fused candidates with a cross-encoder (Section 6)
    # for much higher-precision final ordering. Cross-encoders are too slow
    # to run on millions of docs, but perfectly fine on ~40-80 candidates.
    # -------------------------------------------------------------------
    ranked = cross_encoder_rerank(question, candidates, top_k=40)

    # -------------------------------------------------------------------
    # Step 5: CONFIDENCE GATE #1 -- if even the BEST passage scores too low,
    # we don't have real evidence to answer with. Refuse rather than guess.
    # -------------------------------------------------------------------
    if not any(passage.score > 0.4 for passage in ranked):
        return refused_response("low_retrieval_confidence")

    # -------------------------------------------------------------------
    # Step 6: Take only the top 5 passages -- this keeps the LLM's context
    # small, focused, and less prone to the "lost in the middle" problem
    # (Section 6) that hurts long, noisy contexts.
    # -------------------------------------------------------------------
    top_passages = ranked[:5]
    prompt = build_prompt(question, top_passages)

    # -------------------------------------------------------------------
    # Step 7: Ask the LLM to generate an answer, GROUNDED ONLY in top_passages.
    # temperature=0 for maximum determinism/factuality (no creative sampling).
    # -------------------------------------------------------------------
    raw_answer = llm_generate(prompt, temperature=0)

    # -------------------------------------------------------------------
    # Step 8: CONFIDENCE GATE #2 -- the prompt instructs the LLM to say
    # "INSUFFICIENT_CONTEXT" if the retrieved passages don't actually answer
    # the question. If it says so, we honor that and refuse cleanly.
    # -------------------------------------------------------------------
    if raw_answer.startswith("INSUFFICIENT_CONTEXT"):
        return refused_response("no_relevant_passages")

    # -------------------------------------------------------------------
    # Step 9: CLAIM EXTRACTION + FACT-CHECKING -- this is the actual
    # "zero hallucinations" enforcement mechanism. Break the raw answer
    # into small, individually-checkable "atomic claims" (e.g. one claim
    # per sentence or per fact), then verify EACH claim is actually
    # entailed (logically supported) by the retrieved passages using an
    # NLI (Natural Language Inference) model -- a plain-English one-liner:
    # "a model that decides whether a piece of text logically follows from,
    # contradicts, or is unrelated to another piece of text."
    # If ANY claim can't be verified, we refuse rather than let an
    # unsupported claim slip through to the user.
    # -------------------------------------------------------------------
    claims = extract_atomic_claims(raw_answer)
    for claim in claims:
        if not nli_entailed(claim, top_passages):
            return refused_response("unverified_claim")

    # -------------------------------------------------------------------
    # All gates passed: the answer is grounded, confident, and every claim
    # has been individually verified against the retrieved evidence.
    # -------------------------------------------------------------------
    return RespondResult(answer=raw_answer, passages=top_passages, refused=False)


def refused_response(reason: str) -> RespondResult:
    """A first-class, explicit refusal -- NOT a vague hedge buried in text."""
    return RespondResult(answer="", passages=[], refused=True, refusal_reason=reason)


# NOTE: embed(), qdrant_ann_search(), elasticsearch_bm25(), rrf_fusion(),
# cross_encoder_rerank(), build_prompt(), llm_generate(), extract_atomic_claims(),
# and nli_entailed() are each real functions you'd implement using the
# techniques from Sections 4, 6, and standard NLI models (e.g. a model
# fine-tuned on MNLI/SNLI for entailment classification) -- they are stubbed
# here to keep the read-path LOGIC the star of this example.

9.5 Code: a simple hierarchical chunking function (sentence-boundary + token overlap)

The write path (S3 → chunk ingestor → hierarchical chunking → embeddings → storage) needs a chunker. The notes specify sentence-level boundaries (using spaCy) plus 32-token overlap between consecutive chunks (overlap ensures a sentence that would otherwise get awkwardly split across two chunks still has enough surrounding context in at least one of them).

# pip install spacy --break-system-packages
# python -m spacy download en_core_web_sm
import spacy

nlp = spacy.load("en_core_web_sm")


def chunk_document(text: str, max_tokens_per_chunk: int = 256, overlap_tokens: int = 32) -> list[dict]:
    """
    Hierarchical, sentence-boundary-respecting chunking with token overlap.

    Why sentence boundaries: splitting mid-sentence produces chunks that are
    semantically broken and embed poorly. Why overlap: a fact stated right at
    a chunk boundary shouldn't lose its surrounding context in EVERY chunk
    that references it.
    """
    doc = nlp(text)
    sentences = [sent.text.strip() for sent in doc.sents]

    chunks = []
    current_chunk_sentences: list[str] = []
    current_token_count = 0
    char_cursor = 0

    def rough_token_count(s: str) -> int:
        # A simple approximation (real systems use a proper tokenizer, e.g.
        # tiktoken, matching whatever embedding model will consume the chunk).
        return len(s.split())

    for sentence in sentences:
        sentence_tokens = rough_token_count(sentence)

        # If adding this sentence would overflow the chunk, close the
        # current chunk out and start a new one, carrying over the last
        # `overlap_tokens` worth of sentences for continuity.
        if current_token_count + sentence_tokens > max_tokens_per_chunk and current_chunk_sentences:
            chunk_text = " ".join(current_chunk_sentences)
            chunks.append({
                "text": chunk_text,
                "token_count": current_token_count,
                "char_start": char_cursor,
                "char_end": char_cursor + len(chunk_text),
            })
            char_cursor += len(chunk_text)

            # Build the overlap: keep trailing sentences until we've got
            # roughly `overlap_tokens` worth, to seed the next chunk.
            overlap_sentences, overlap_count = [], 0
            for sent in reversed(current_chunk_sentences):
                overlap_sentences.insert(0, sent)
                overlap_count += rough_token_count(sent)
                if overlap_count >= overlap_tokens:
                    break

            current_chunk_sentences = overlap_sentences
            current_token_count = overlap_count

        current_chunk_sentences.append(sentence)
        current_token_count += sentence_tokens

    # Flush the final chunk
    if current_chunk_sentences:
        chunk_text = " ".join(current_chunk_sentences)
        chunks.append({
            "text": chunk_text,
            "token_count": current_token_count,
            "char_start": char_cursor,
            "char_end": char_cursor + len(chunk_text),
        })

    return chunks


# --- Example ---------------------------------------------------------------
sample_doc = (
    "Throughput degradation under sustained load is a common issue. "
    "It is typically caused by connection pool exhaustion. "
    "Another frequent cause is unindexed database queries. "
    "Monitoring dashboards should track p50, p95, and p99 latency. "
    "Alerts should fire when p99 crosses a defined SLO threshold."
)

for i, chunk in enumerate(chunk_document(sample_doc, max_tokens_per_chunk=15, overlap_tokens=5)):
    print(f"Chunk {i}: {chunk['text']}")

Each resulting chunk (a "passage" in the notes' schema) would then get an embedding computed (using one of the models the notes list — microsoft/hamier, Qwen3 (8B), BGE-M3, or an OpenAI/OSS embedding API), stored in Qdrant (the vector), Postgres (the metadata: doc_id, chunk_index, char_start, char_end, token_count), and ElasticSearch (the BM25 inverted index).

9.6 The write path, explained in words

Flow: S3 (raw doc storage) → Chunk Ingestor → Hierarchical Chunking → Embeddings → Storage (Postgres, Qdrant, ElasticSearch, Redis).

Step by step: a document lands in S3. A chunk ingestor process picks it up (typically via an event notification or polling), performs three sub-steps the notes list explicitly (list files, download, read), then runs hierarchical chunking (Section 9.5's sentence-boundary + overlap logic) to split it into passages. Each passage is embedded, and the results are fanned out to storage: Postgres gets the passage/doc metadata, Qdrant gets the embedding vectors, and ElasticSearch gets the text for its BM25 inverted index. This asynchronous pipeline is exactly why the system can tolerate the ~15-minute eventual consistency SLO mentioned earlier — the write path doesn't need to be instantaneous, it needs to be reliable and complete within a bounded, acceptable delay.

The notes also flag a timebox mechanism on the read side with two named fallback behaviors — cascade and fallback — meaning if some part of the read path takes too long (e.g., the cross-encoder rerank step or the LLM generation call, described as "the long pole" of end-to-end latency), the system has a time budget after which it cascades to a simpler/cheaper strategy or falls back to a safe default (like a refusal) rather than hanging indefinitely. This connects directly back to the query latency numbers given in the notes: p50 = 1.5 sec, p99 = 5 sec, with the LLM inference call explicitly called out as the long pole (the slowest, bottleneck stage) — largely because the very first token the user sees (their Time To First Token, covered next in the Appendix) is gated on that call finishing its reasoning.

10. Appendix A: Streaming Tool Calls

10.1 The simple example first

Think about the difference between getting a text message that appears all at once after a long pause, versus watching someone type it live, word by word, right in front of you. Even though the total message might take the same amount of time to fully arrive, watching it appear progressively feels far less frustrating — you get immediate feedback that something is happening, instead of staring at a blank screen wondering if anything is working at all.

That's the entire motivation for streaming: even if the total response time is identical, showing partial output as it's generated dramatically improves perceived responsiveness.

10.2 Technical explanation

The most important metric here, per the notes: TTFT — Time To First Token (a plain-English one-liner: "how long the user waits before they see ANY output at all, even if the full response isn't done yet"). If a tool call takes real time to execute (a slow database query, a slow external API), and the user has to wait through the entire tool call before seeing anything, that's flagged directly in the notes as bad UX — "like everything else," slow, silent waiting erodes trust in the product.

The tricky part specific to streaming tool calls: "what if tool input arrives as partial JSON fragments?" When a model streams its response token by token, and part of that response is a tool_use block (e.g., building up {"city": "Tok ... yo"} character by character), you cannot execute the tool with a half-formed, syntactically-invalid JSON blob. The notes' answer: "wait until the entire input is available" — i.e., keep buffering (accumulating) the partial JSON text as it streams in, and only attempt to parse/execute the tool call once the model signals that this particular content block is fully complete.

The notes include a cautionary, very concrete illustration of why this matters: imagine the model is streaming a dangerous tool call like "delete that record..." If your system tried to act on a half-baked, incomplete tool input (say, a delete call where the id argument hasn't fully arrived yet), it could either error out unpredictably or — worse — "no input could be fired" i.e., act on a malformed/empty argument. Half-baked input / tool calls with no input could be fired is exactly the failure mode the buffering strategy exists to prevent.

10.3 Code: the notes' Anthropic SDK streaming example, cleaned up and explained line by line

import anthropic
import json

client = anthropic.Anthropic()

# The tool definition is identical in shape to Section 1 -- streaming
# doesn't change WHAT a tool looks like, only HOW the response arrives.
tools = [{
    "name": "get_weather",
    "description": "Get current weather for a location",
    "input_schema": {
        "type": "object",
        "properties": {
            "location": {"type": "string"},
            "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]},
        },
        "required": ["location"],
    },
}]


def execute_tool(tool_name: str, tool_input: dict) -> dict:
    """Stand-in for a real tool implementation (see Section 1 for a fuller mock)."""
    if tool_name == "get_weather":
        return {"location": tool_input["location"], "temp": 24, "condition": "sunny"}
    return {"error": "unknown tool"}


with client.messages.stream(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    tools=tools,
    messages=[{"role": "user", "content": "What is the weather in Tokyo?"}],
) as stream:

    # These two variables ACCUMULATE state across streamed events -- this IS
    # the "buffering" the notes insist on. We never act on partial state.
    current_tool_input = ""
    current_tool_name = None

    for event in stream:
        # -------------------------------------------------------------
        # "content_block_start": a NEW block (either plain text or a
        # tool_use block) has begun streaming. If it's a tool_use block,
        # note its NAME immediately, but its INPUT starts empty --
        # arguments arrive incrementally via later delta events.
        # -------------------------------------------------------------
        if event.type == "content_block_start":
            if event.content_block.type == "tool_use":
                current_tool_name = event.content_block.name
                current_tool_input = ""  # reset the buffer for this new block

        # -------------------------------------------------------------
        # "content_block_delta": an incremental CHUNK of the current
        # block has arrived. For tool_use blocks, this is a fragment of
        # partial JSON (event.delta.type == "input_json_delta") -- we
        # must ACCUMULATE it, not parse it yet, since it's incomplete.
        # For plain text blocks, we can print immediately since partial
        # text is still readable (this is what gives the "word by word"
        # streaming feel for normal chat responses).
        # -------------------------------------------------------------
        elif event.type == "content_block_delta":
            if event.delta.type == "input_json_delta":
                current_tool_input += event.delta.partial_json  # <- Accumulate
            elif event.delta.type == "text_delta":
                print(event.delta.text, end="", flush=True)

        # -------------------------------------------------------------
        # "content_block_stop": the CURRENT block (text or tool_use) has
        # finished streaming completely. If it was a tool_use block, the
        # buffered JSON string is now guaranteed complete and valid --
        # THIS is the safe moment to parse it and actually execute the
        # tool, never before.
        # -------------------------------------------------------------
        elif event.type == "content_block_stop":
            if current_tool_name:
                parsed_input = json.loads(current_tool_input)  # <- input complete
                result = execute_tool(current_tool_name, parsed_input)  # <- execute
                print(f"\n[Executed {current_tool_name} -> {result}]")
                current_tool_name = None  # reset for the next potential tool block

The three event types to remember: content_block_start (a new block began — note its type/name), content_block_delta (a fragment arrived — accumulate if it's tool JSON, print immediately if it's plain text), and content_block_stop (the block is fully complete — only now is it safe to parse and execute a tool call).

11. Appendix B: Evaluating RAG with RAGAS

11.1 The simple example first

Context Precision, simple example: think of a search-results page with 5 blue links. Some of them are genuinely useful and directly relevant to what you searched for (imagine green dots next to them); others are irrelevant filler or noise (red dots). A good search engine doesn't just include some relevant results somewhere on the page — it puts them near the top. A search engine that buries the one truly relevant result at position #5, behind four irrelevant ones, is worse than one that puts it at position #1, even if both technically "found" it somewhere on the page.

Faithfulness, simple example: imagine grading a student's essay by checking, sentence by sentence, whether each individual claim they made is actually backed up by the assigned textbook. If the student writes five sentences and four of them are directly supported by the textbook but the fifth one is something they simply made up (that isn't in the textbook at all), you wouldn't give them full credit — you'd score the essay based on the fraction of sentences that are actually grounded in the source material. That's exactly what "faithfulness" measures for an LLM's generated answer.

11.2 Technical explanation

The motivation, per the notes: naive RAG evaluation is just qualitative — "does it seem right?" — and manual feedback loops are slow. That doesn't scale, and it's subjective. RAGAS (a plain-English one-liner: "a framework and library for evaluating RAG pipelines using automated, LLM-assisted metrics instead of manual human judgment") fixes two specific pain points named in the notes: no need for human-annotated ground truth, and it lets you verify that changes/optimizations to your pipeline don't quietly downgrade quality (a regression-testing mindset applied to RAG quality, not just code correctness).

Context Precision — measures the retriever's (or reranker's) ability to rank relevant chunks higher, and it directly penalizes systems that bury relevant content below noise — this is precisely the "lost in the middle" problem flagged back in Section 6. The method: use an LLM to label each retrieved chunk as RELEVANT or NOT relevant, and then, for every position k in the ranked list, compute precision@k — the fraction of the top k results that are relevant.

The notes' worked example, using a 5-result list marked (green = relevant, red = not relevant): 🟢 🔴 🟢 🔴 🔴

p@1 = 1 — the top 1 result: 1 out of 1 is relevant → 1/1 = 1.
p@2 = 0 — wait, this looks odd at first: the notes explicitly write p@2 = 0. Reading the notes' own math carefully: precision@k is only counted (included in the final average) at positions where the k-th result itself is relevant — i.e., you only "credit" a position if the newly added result at that rank is itself a hit, otherwise that position contributes 0 to the running average (this is the standard "average precision" style computation, where precision at non-relevant ranks doesn't get counted toward the final score).
p@3 = 2/3 — of the top 3 results, 2 are relevant (the 1st and 3rd) → 2/3 ≈ 0.67.
p@4 = 0, p@5 = 0 — same logic: the 4th and 5th results are themselves not relevant, so they don't contribute.

Final Context Precision formula (from the notes' own arithmetic):

Context Precision = (p@1 + p@3) / (# relevant chunks)
                   = (1 + 0.67) / 2
                   = 1.67 / 2
                   = 0.835

In words: you sum up the precision@k values at each rank where a relevant chunk appears, then divide by the total number of relevant chunks in the list (here, 2 relevant chunks out of 5 total). This rewards systems that put their relevant chunks early — a relevant chunk found at rank 1 contributes a full 1.0, while one found deeper in the list contributes a smaller fraction, exactly reflecting the "don't bury the good stuff" intuition from the simple example.

Faithfulness Score — measures factual consistency of the generated answer, and is explicitly called a "measure of hallucination" in the notes. Method: break the answer into atomic claims (using an LLM — the same "claim extraction" step from the read-path pseudocode in Section 9.4), then for each claim, score it 1 if the retrieved context supports the claim, or 0 if the context does not support the claim.

Faithfulness formula:

faithfulness = (# claims supported) / (# claims)

This is the exact same underlying idea as the nli_entailed() check baked into the "zero hallucinations" respond() function from Section 9 — faithfulness is essentially the evaluation-time metric version of the same runtime guardrail.

11.3 Code: Context Precision and Faithfulness, computed from scratch

# =============================================================================
# CONTEXT PRECISION -- from a list of relevant/irrelevant flags
# (the "green dot / red dot" example from the notes).
# =============================================================================

def context_precision(relevance_flags: list[bool]) -> float:
    """
    relevance_flags: e.g. [True, False, True, False, False]
                      -- True = relevant chunk, False = not relevant,
                      in the ORDER they were retrieved/ranked (best first).

    Returns the RAGAS-style Context Precision score.
    """
    total_relevant = sum(relevance_flags)
    if total_relevant == 0:
        return 0.0  # no relevant chunks at all -- precision is undefined/zero

    running_relevant_count = 0
    precision_sum = 0.0

    for k, is_relevant in enumerate(relevance_flags, start=1):
        if is_relevant:
            running_relevant_count += 1
            # precision@k = (# relevant chunks in top k) / k
            precision_at_k = running_relevant_count / k
            precision_sum += precision_at_k
        # if NOT relevant, this rank contributes 0 and is skipped, matching
        # the notes' worked example where p@2, p@4, p@5 = 0

    return precision_sum / total_relevant


# --- Example matching the notes exactly: 🟢 🔴 🟢 🔴 🔴 ----------------------
flags = [True, False, True, False, False]
score = context_precision(flags)
print(f"Context Precision: {score:.3f}")  # -> should print 0.835


# =============================================================================
# FAITHFULNESS -- given a list of claims and whether each is supported.
# =============================================================================

def faithfulness_score(claim_support_labels: list[bool]) -> float:
    """
    claim_support_labels: e.g. [True, True, True, False]
                           -- True = context supports this atomic claim,
                           False = context does NOT support it (hallucinated).
    """
    if not claim_support_labels:
        return 1.0  # no claims made -- vacuously faithful (nothing to check)

    supported = sum(claim_support_labels)
    total = len(claim_support_labels)
    return supported / total


# --- Example: an answer with 4 atomic claims, 1 of which is unsupported ----
claims_supported = [True, True, True, False]
faith_score = faithfulness_score(claims_supported)
print(f"Faithfulness: {faith_score:.3f}")  # -> 0.75

11.4 Code: using the real RAGAS library

# pip install ragas datasets --break-system-packages
from ragas import evaluate
from ragas.metrics import context_precision, faithfulness
from datasets import Dataset

# RAGAS expects a dataset shaped like this: one row per question, with the
# retrieved contexts and the generated answer already filled in.
eval_dataset = Dataset.from_dict({
    "question": ["Why is my API returning a 500 error under high load?"],
    "answer": ["500 errors under high load are typically caused by connection "
               "pool exhaustion or unhandled exceptions from resource contention."],
    "contexts": [[
        "Throughput degradation under sustained load can trigger unhandled "
        "exceptions, which surface to callers as HTTP 500 errors.",
        "Connection pool exhaustion is a common cause of request failures "
        "under high concurrency.",
    ]],
    # ground_truth is optional for some metrics but required for others (e.g. recall)
    "ground_truth": ["500 errors under load are usually caused by connection "
                     "pool exhaustion or unhandled exceptions."],
})

results = evaluate(eval_dataset, metrics=[context_precision, faithfulness])
print(results)
# This runs the SAME conceptual computation as the from-scratch functions
# above, but uses an LLM internally to (a) judge chunk relevance for
# context_precision, and (b) extract + verify atomic claims for faithfulness.

12. Glossary

Tool use / function calling — letting an LLM request that your application run a specific, predefined function on its behalf, using arguments the model constructs.
Tool schema — the JSON description (name, description, parameters) that tells the model what a tool does and how to call it; the model never sees or runs the actual code.
Enum — a parameter constrained to a fixed list of valid values, so a model can't invent arbitrary strings.
RBAC (Role-Based Access Control) — restricting what a user can see or do based on their assigned role (e.g., department, seniority).
Dense retrieval — semantic search using embeddings; finds documents with similar meaning, even without shared words.
Sparse retrieval / BM25 — traditional keyword-based search; scores documents by exact term overlap, weighted by term rarity/frequency.
Hybrid search — running dense and sparse retrieval together and merging the results, to cover both of their blind spots.
RRF (Reciprocal Rank Fusion) — a method for merging multiple ranked lists into one, based only on each item's rank position in each list, not its raw score.
Metadata filtering — applying hard constraints (date, department, tags) before or after a search, on top of relevance ranking.
Pre-filtering — applying metadata constraints inside the search query itself, so the database never considers disallowed documents.
Post-filtering — running an unfiltered search first, then discarding invalid results afterward in application code.
Bi-encoder — embeds the query and each document independently, then compares them via vector similarity; fast and scalable but can't compare them in context.
Cross-encoder — feeds the query and a single document together as one input, producing one precise relevance score; accurate but slow and non-scalable.
"Lost in the middle" problem — the tendency for information in the middle of a long context to receive less effective attention than information at the start or end.
HyDE (Hypothetical Document Embedding) — generating a fake, ideal-sounding answer to a query first, then using that text's embedding for retrieval, to bridge a vocabulary gap between casual queries and formal documents.
Semantic caching — caching LLM responses keyed by meaning (embedding similarity) rather than exact text match, so paraphrased questions can still hit the cache.
NLI (Natural Language Inference) — a model task/technique for deciding whether one piece of text logically follows from (entails), contradicts, or is unrelated to another.
HNSW (Hierarchical Navigable Small World) — a graph-based index structure used by vector databases to find approximately-nearest vectors quickly without comparing against every single one.
TTFT (Time To First Token) — how long a user waits before seeing any output at all, even if the full response is still being generated.
RAGAS — a framework/library for evaluating RAG pipelines with automated, LLM-assisted metrics instead of manual human review.
Context Precision — a RAGAS metric measuring whether relevant retrieved chunks are ranked near the top rather than buried lower down.
Faithfulness — a RAGAS metric measuring what fraction of an LLM's generated claims are actually supported by the retrieved context (a direct measure of hallucination).

13. "Why This Matters in Production" — One-Liner Per Section

Tool Use Fundamentals — without a disciplined intercept/execute/return loop, you either let a model hallucinate data it can't actually access, or you build a system that silently breaks the moment a tool call needs real-world side effects.
Tool Schema Design — a vague schema is invisible in your own testing and only breaks in production, when real users ask messy questions your clean test prompts never covered.
Parallel Tool Calls and Failure Handling — sequential tool calls make your product feel painfully slow at scale, and having no explicit failure strategy means one flaky dependency can silently corrupt or crash every response.
Hybrid Search — relying on dense-only or sparse-only retrieval guarantees you'll systematically miss either exact-term queries or natural-language queries, and users will experience this as "the search is broken" without knowing why.
Metadata Filtering — get pre- vs post-filtering wrong and you either waste retrieval budget or, worse, leak data across security boundaries like RBAC in enterprise settings.
Reranking (Cross-Encoders vs Bi-Encoders) — skipping reranking means your final answer is only as good as noisy first-pass retrieval; running cross-encoders on your whole corpus means your product times out.
Query Rewriting with HyDE — without bridging the query-document vocabulary gap, your retrieval will keep failing on exactly the domains (legal, scientific, internal jargon) where correctness matters most.
Semantic Caching — every uncached LLM call costs real latency and real money; miscalibrating the threshold either serves wrong answers or barely saves anything at all.
Full RAG System Design — at 10M-document scale, "correctness" has to be engineered in at every stage (retrieval, reranking, generation, fact-checking), because a single loose stage can let an unsupported claim reach the user.
Streaming Tool Calls — users abandon or distrust products that feel frozen; but streaming without proper buffering risks executing dangerous operations (like deletes) on incomplete input.
Evaluating RAG with RAGAS — without automated, repeatable metrics, you can't tell whether a "quality improvement" you shipped actually helped or quietly made hallucinations worse.

Your Email List Is Decaying Right Now

MailValid.io — Sun, 12 Jul 2026 18:11:13 +0000

Verifying an email address at signup only proves it was valid at that moment. It says nothing about whether it's still valid six months later and for most contact databases, a meaningful share of it won't be. If your app verifies once at signup and never again, you're shipping a slow leak, not a fix.

This post covers why lists decay even without a single unsubscribe, what a re-verification job should actually check, and a basic cron-based implementation for keeping a contacts table clean over time.

Why valid-at-signup doesn't mean valid-forever

Email addresses have a shelf life, and email verification once at collection time only confirms the address was live on that specific day. Roughly 22.5% of email contacts become invalid over the course of a year — around 2.1% per month: driven by job changes, account deactivation, provider migrations, and abandoned inboxes. B2B contact lists decay faster than consumer lists, commonly in the 25–30% annual range, largely because employees change jobs and their work addresses get deactivated almost immediately.

If your database has contacts older than a few months and you've never re-verified, you're very likely sending to a nontrivial number of dead addresses without knowing it.

What causes Email list decay

Email list decay is driven by five recurring patterns, and only one of them is under your control.

Job changes — the single biggest driver for B2B lists; an estimated 70% of B2B job-related email addresses change within 12 months of collection.
Company changes — mergers, acquisitions, rebrands, or shutdowns can invalidate an entire domain's worth of addresses at once.
Provider migrations — a contact moves from one email provider to another and abandons the old address.
Account deactivation — mailbox providers periodically purge inactive accounts per their own retention policies.
Signup typos — the one decay source that real-time syntax/MX verification at signup actually prevents.

The first four happen entirely outside your app, on a timeline you don't control, which is exactly why a one-time verification at signup can't be the whole strategy.

Email Spam traps: the risk re-verification catches that bounces don't

A recycled spam trap is a real address that mailbox providers have reclaimed after roughly 12 months of inactivity, then repurposed specifically to catch senders with stale lists. This is the mechanism that makes re-verification more than a bounce-rate optimization — it's a reputation risk mitigation.

There are three recognized spam trap types:

Typo traps - created from signup typos (user@yahooo.com), catching senders who never validated syntax carefully.
Recycled traps - formerly legitimate addresses that went dormant for a year or more and were reclaimed by the provider as a monitoring mechanism.
Pristine traps - addresses that were never real signups at all, created solely to catch senders using purchased or scraped lists.

Recycled traps are the one your own signup-time verification can't catch, because the address was real and valid when the person signed up. It only becomes a trap after months of dormancy — which is precisely the gap a scheduled re-verification job is designed to close.

Designing the re-verification job

A re-verification job needs three things a one-off signup check doesn't:

A cadence - how often each contact gets re-checked (see the how-often section)
Prioritization - verifying your entire table every night at scale is wasteful; segment by last-engagement date or last-verified date and check the stalest/riskiest contacts first
A status write-back, not a delete — store the verification result (valid, invalid, catch-all, disposable, unknown) and last-checked timestamp rather than silently dropping rows, so your sending logic can make the suppress/send decision explicitly

A scheduled job example (Node.js + cron)

javascriptconst cron = require('node-cron');
const db = require('./db');

// Runs nightly at 2am — pulls contacts not verified in 90+ days,
// oldest-first, capped at a batch size to respect rate limits.
cron.schedule('0 2 * * *', async () => {
  const staleContacts = await db.query(`
    SELECT id, email FROM contacts
    WHERE last_verified_at < NOW() - INTERVAL '90 days'
       OR last_verified_at IS NULL
    ORDER BY last_verified_at ASC NULLS FIRST
    LIMIT 5000
  `);

  for (const contact of staleContacts) {
    const result = await verifyEmail(contact.email); // your verification call
    await db.query(`
      UPDATE contacts
      SET verification_status = $1, last_verified_at = NOW()
      WHERE id = $2
    `, [result.status, contact.id]);

    if (result.status === 'invalid') {
      await db.query(`UPDATE contacts SET suppressed = true WHERE id = $1`, [contact.id]);
    }
  }
});

The batch size and interval here are illustrative, tune them to your sending volume and your verification provider's rate limits. The important structural choice is the last_verified_at column: it turns re-verification into an ongoing, queryable process instead of a one-time event.

How often should you re-verify?

There's no universal answer, but the decay data gives a reasonable default: at roughly 2% monthly decay, a 90-day re-verification cycle catches most staleness before it compounds into a meaningful bounce or spam-trap risk, without re-checking addresses that haven't had time to go stale. High-value B2B lists with faster job-change turnover may justify a 30–60 day cycle; low-churn consumer lists can often stretch to 6 months.

If you're re-verifying manually or building this in-house, an email verification API that returns a distinct catch-all status (rather than lumping it in with valid) matters more here than at signup time, a domain's catch-all configuration can change between checks, and treating it as a hard valid inflates your list-quality metrics without actually reducing risk. MailValid supports exactly this: bulk re-verification with status codes designed for exactly this kind of scheduled job, plus a real-time API for the signup-time check.

Frequently Asked Questions

Does re-verification replace signup-time verification?
No, they catch different things. Signup-time verification (syntax + MX + SMTP) prevents typo'd and fake addresses from entering your database at all. Re-verification catches addresses that were genuinely valid at signup but have since gone stale.

Will re-verification hurt my sender reputation the way sending mail does?
No. SMTP-level verification checks mailbox existence via a RCPT TO command without completing delivery (no DATA command is sent), so it doesn't count as a send and doesn't itself generate spam complaints, provided it's run through a provider with dedicated verification IP infrastructure rather than your own sending domain.

Should I delete contacts that fail re-verification?
Suppressing them from sends is safer than deleting outright, a suppressed contact might resubscribe, correct a domain-level issue, or represent a data point you still want for reporting. Deletion is a business decision, not a deliverability requirement.

What if a contact I haven't emailed in a year suddenly becomes a spam trap?
This is exactly the recycled-trap scenario. If your last engagement with an address predates your last verification by many months, treat it as higher risk and prioritize it in your re-verification queue rather than assuming dormancy is harmless.

Web Scraping with Playwright: Handle Any Website

qing — Sun, 12 Jul 2026 18:08:28 +0000

Web Scraping with Playwright: Handle Any Website

Imagine having the ability to extract data from any website, no matter how complex or dynamic the content. This is where web scraping comes in – a powerful technique that allows us to automate the collection of data from websites. However, traditional web scraping methods often struggle with modern websites that employ anti-scraping measures, making it difficult to extract the data we need.

This is where Playwright comes in – a revolutionary tool that has changed the game when it comes to web scraping. Playwright is a Node.js library developed by Microsoft, which provides a high-level API for automating web browsers. In this article, we'll explore how to use Playwright for web scraping, and demonstrate its capabilities with a practical example in Python.

Setting Up Playwright

Before we dive into the code, let's first set up Playwright. If you're using Python, you can install Playwright using pip:

pip install playwright

You'll also need to install the wpt package, which is used to drive the browser:

pip install wpt

Once installed, you can verify that Playwright is working by running the following code:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://www.example.com")
    browser.close()

This code launches a new Chromium browser instance, navigates to the specified URL, and then closes the browser.

Handling Dynamic Content with Playwright

One of the biggest challenges when it comes to web scraping is handling dynamic content. This can include elements that are loaded after the initial page load, or content that is generated on the fly using JavaScript. Playwright makes it easy to handle these types of elements by providing a powerful API for interacting with web pages.

Let's take a look at an example code snippet that demonstrates how to handle dynamic content with Playwright:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://www.example.com")

    # Wait for the page to load
    page.wait_for_load_state("networkidle0")

    # Get all the elements with the class "dynamic-element"
    elements = page.query_selector_all(".dynamic-element")

    # Iterate over the elements and print their text content
    for element in elements:
        print(element.text_content())

    browser.close()

In this code snippet, we use the wait_for_load_state method to wait for the page to load, and then use the query_selector_all method to get all elements with the class "dynamic-element". We then iterate over the elements and print their text content.

Handling Anti-Scraping Measures

Modern websites often employ anti-scraping measures to prevent web scraping. These can include CAPTCHAs, rate limiting, and other techniques designed to make it difficult for bots to access the website. Playwright provides a number of features that make it easy to handle these types of anti-scraping measures.

One of the most powerful features of Playwright is its ability to solve CAPTCHAs using the Google CAPTCHA API. To use this feature, you'll need to install the google-captcha package:

pip install google-captcha

You can then use the following code snippet to solve a CAPTCHA:

from googlecaptcha import GoogleCaptcha
from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch()
    page = browser.new_page()
    page.goto("https://www.example.com")

    # Get the CAPTCHA image
    captcha_image = page.query_selector(".captcha-image").get_attribute("src")

    # Solve the CAPTCHA
    captcha_solver = GoogleCaptcha()
    captcha_token = captcha_solver.solve(captcha_image)

    # Enter the CAPTCHA token on the page
    page.fill("captcha-field", captcha_token)

    browser.close()

Handling Rate Limiting

Rate limiting is another common anti-scraping measure that can make it difficult to access a website. Playwright provides a number of features that make it easy to handle rate limiting.

One of the most powerful features of Playwright is its ability to simulate a human user. By default, Playwright simulates a human user by introducing delays between requests. This makes it difficult for the website to detect that you're a bot.

However, if you need to make a large number of requests to a website that employs rate limiting, you may need to take additional steps to simulate a human user. One way to do this is to use a rotating proxy, which can help to distribute your requests across multiple IP addresses.

Conclusion

In this article, we've explored how to use Playwright for web scraping. We've demonstrated its capabilities with a practical example in Python, and shown how to handle dynamic content and anti-scraping measures.

Whether you're a seasoned web scraper or just starting out, Playwright is a powerful tool that can help you extract data from any website. With its powerful API and features like CAPTCHA solving and rotating proxies, Playwright makes it easy to handle even the most complex websites.

So why not give Playwright a try today? With its ease of use and powerful features, it's the perfect tool for any web scraper.

💡 Related: **Content Creator Ultimate Bundle (Save 33%)* — $29.99*

إعداد بيئة تطوير آمنة باستخدام نظام لينكس واداة الدッカー لتشغيل تطبيقات الويب بشكل متوازن وآمن

Issam Hilmi — Sun, 12 Jul 2026 18:05:20 +0000

إعداد بيئة تطوير آمنة باستخدام نظام لينكس واداة الدッカー
إعداد بيئة تطوير آمنة باستخدام نظام لينكس واداة الدッカー لتشغيل تطبيقات الويب بشكل متوازن وآمن هو موضوع هام في عالم التطوير البرمجي، حيث أن الأمان يعد العامل الأكثر أهمية في بناء التطبيقات.
لماذا نظام لينكس؟
نظام لينكس هو نظام تشغيل مفتوح المصدر ويُعتبر أكثر الأنظمة أمانًا، كما أنه يدعم العديد من الأدوات التي تساعد في بناء بيئة تطوي

Originally published on my blog.

This article was originally published on my blog and is republished here with permission.

Nginx + Python: The Perfect Production Setup

qing — Sun, 12 Jul 2026 18:05:17 +0000

Nginx + Python: The Perfect Production Setup

Why You Need Nginx and Python in Your Production Setup

As a developer, there are few things more satisfying than deploying a beautiful Python application to production. But let's be real – getting it to production is only half the battle. The real challenge begins when you need to ensure your application is secure, scalable, and performs well under load.

That's where Nginx comes in – the lightweight, powerful web server that's been a staple of production environments for years. And when paired with Python, the results are nothing short of magic.

In this post, we'll explore the perfect production setup for your Python application: Nginx + Python. We'll cover the benefits of using Nginx, how to set it up with Python, and provide a working example to get you started.

Why Nginx?

So, why do you need Nginx in your production setup? Here are just a few reasons:

High-performance: Nginx is known for its lightning-fast performance, even under heavy load. This makes it the perfect choice for applications that require a high level of concurrency.
Lightweight: Unlike other web servers like Apache, Nginx is incredibly lightweight, making it easy to deploy and manage.
Security: Nginx includes built-in security features like SSL/TLS support, HTTP/2 support, and WAF (Web Application Firewall) capabilities.
Flexibility: Nginx can be used as a reverse proxy, load balancer, and caching layer, making it a versatile addition to your production setup.

Setting Up Nginx with Python

So, how do you set up Nginx with Python? Here's a step-by-step guide:

Step 1: Install Nginx and Python

First, you'll need to install Nginx and Python on your server. You can do this using your distribution's package manager or by compiling from source.

sudo apt-get update
sudo apt-get install nginx python3

Step 2: Create a New Virtual Host

Next, create a new virtual host for your Python application. You can do this by creating a new file in the /etc/nginx/sites-available directory, for example:

sudo nano /etc/nginx/sites-available/python_app

Step 3: Configure Nginx

In the new file, add the following configuration:

server {
    listen 80;
    server_name yourdomain.com;

    location / {
        proxy_pass http://localhost:5000;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
    }
}

Replace yourdomain.com with your actual domain name, and 5000 with the port number you're using for your Python application.

Step 4: Activate the New Virtual Host

Next, create a symbolic link to the new virtual host file in the /etc/nginx/sites-enabled directory:

sudo ln -s /etc/nginx/sites-available/python_app /etc/nginx/sites-enabled/

Step 5: Reload Nginx

Finally, reload Nginx to apply the changes:

sudo service nginx reload

Working Example: A Simple Flask Application

Now that we've set up Nginx with Python, let's create a simple Flask application to test it out. Here's an example:

from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/data', methods=['GET'])
def get_data():
    return jsonify({'data': 'Hello, World!'})

if __name__ == '__main__':
    app.run(debug=True, port=5000)

Save this code to a file called app.py and run it using python3 app.py. Then, open your web browser and navigate to http://yourdomain.com/data. You should see the message "Hello, World!".

Conclusion

In this post, we explored the perfect production setup for your Python application: Nginx + Python. We covered the benefits of using Nginx, how to set it up with Python, and provided a working example to get you started.

Whether you're building a high-traffic web application or a simple REST API, Nginx + Python is the perfect combination for any production environment. So why wait? Get started today and experience the power of Nginx + Python for yourself.

Call to Action

Ready to take your Python application to the next level? Try out Nginx + Python today and see the difference for yourself. Share your experiences and tips in the comments below, and don't forget to follow us for more technical blog posts and tutorials!

Build a Real-Time Dashboard with Python and WebSockets

qing — Sun, 12 Jul 2026 18:04:24 +0000

Build a Real-Time Dashboard with Python and WebSockets

Building a Real-Time Dashboard with Python and WebSockets

Imagine having a dashboard that updates in real-time, reflecting the current state of your system or application. No more refreshing the page or waiting for delayed updates - your dashboard is always up-to-date, providing you with instant insights into what's happening. This is the power of real-time data, and it's easier to achieve than you think.

What are WebSockets?

To understand how we'll build a real-time dashboard, we need to talk about WebSockets. WebSockets are a bi-directional communication protocol that allows your web application to send and receive data in real-time. Unlike traditional HTTP requests, WebSockets establish a persistent connection between the client and server, enabling efficient, low-latency communication.

Here's how it works: when a client (usually a web browser) establishes a WebSocket connection, it sends a request to the server to initiate the connection. The server then accepts the connection and sends a response back to the client, indicating that the connection is established. From this point forward, both the client and server can send data to each other without having to make a new request.

Setting Up a WebSocket Server with Python

To build our real-time dashboard, we'll use the websockets library in Python. This library provides a simple and efficient way to establish WebSocket connections and handle real-time data.

First, we'll install the websockets library using pip:

pip install websockets

Next, let's create a basic WebSocket server using Python:

import asyncio
import websockets

async def handle_connection(websocket):
    while True:
        try:
            # Wait for incoming messages from the client
            message = await websocket.recv()
            print(f"Received message: {message}")
            # Send a response back to the client
            await websocket.send(f"Server received your message: {message}")
        except websockets.ConnectionClosed:
            print("Connection closed")
            break

async def main():
    async with websockets.serve(handle_connection, "localhost", 8765) as server:
        print(f"Server listening on port 8765")
        await server.wait_closed()

if __name__ == "__main__":
    asyncio.run(main())

This code sets up a WebSocket server that listens for incoming connections on port 8765. When a client connects, the server prints out any incoming messages and sends a response back to the client.

Creating a Real-Time Dashboard with WebSocket Clients

Now that we have a WebSocket server up and running, let's create a simple dashboard client using JavaScript and the HTML5 WebSocket API.

We'll use the socket.io-client library to establish a WebSocket connection to our server:

<!DOCTYPE html>
<html>
<head>
  <title>Real-Time Dashboard</title>
  <script src="https://cdn.jsdelivr.net/npm/socket.io-client@2/dist/socket.io.js"></script>
  <script>
    // Establish a WebSocket connection to our server
    var socket = io("ws://localhost:8765");

    // Update the dashboard with incoming data
    socket.on("message", function(message) {
      console.log(message);
      document.getElementById("dashboard").innerHTML += `<p>${message}</p>`;
    });
  </script>
</head>
<body>
  <h1>Real-Time Dashboard</h1>
  <div id="dashboard"></div>

  <script>
    // Send a message to the server
    document.getElementById("sendButton").addEventListener("click", function() {
      var message = document.getElementById("messageInput").value;
      socket.emit("message", message);
    });
  </script>
</body>
</html>

This code establishes a WebSocket connection to our server and updates a dashboard element with any incoming messages. We also have a send button that sends a message to the server when clicked.

Putting It All Together

To see our real-time dashboard in action, we'll run both the server and client code in different terminal windows. First, let's start the server:

python server.py

Next, open a web browser and navigate to http://localhost:8765. This will establish a WebSocket connection to our server, and we should see the dashboard update in real-time.

To test the real-time functionality, let's send a message to the server from the client code. Enter a message in the input field and click the send button. The dashboard should update immediately with the message we just sent.

Conclusion

Building a real-time dashboard with Python and WebSockets is easier than you think. By using the websockets library in Python and establishing a WebSocket connection to our server, we can create a dashboard that updates in real-time. With this knowledge, you can build your own real-time dashboards, reflecting the current state of your system or application.

So, what are you waiting for? Get started today and create your own real-time dashboards with Python and WebSockets!

Python Virtual Environments: A Complete 2025 Guide

qing — Sun, 12 Jul 2026 18:03:40 +0000

Python Virtual Environments: A Complete 2025 Guide

The Problem with Global Packages

When you're working on a project in Python, you'll often find yourself installing packages using pip, the Python package manager. But have you ever stopped to think about what happens when you install a package globally? The Python interpreter looks for packages in its sys.path list, which includes the current working directory and any directories specified by the PYTHONPATH environment variable. But what if you're working on multiple projects at the same time, each with its own dependencies? Installing packages globally can lead to conflicts between projects, and it's a nightmare to manage.

The Solution: Virtual Environments

That's where virtual environments come in. A virtual environment is a self-contained Python environment that allows you to install packages without affecting the global Python environment. You can think of it like a sandbox where you can experiment with different versions of Python and packages without breaking anything.

History of Virtual Environments

Virtual environments have been around since the early days of Python. The first virtual environment tool was virtualenv, which was released in 2006. virtualenv was a great tool, but it had some limitations, such as not being able to handle packages with dependencies on other packages. To address this, virtualenvwrapper was released in 2008, which added some useful features, such as being able to create and manage multiple virtual environments.

In 2014, virtualenv was replaced by virtualenv-20, which is now known as venv. venv is the built-in virtual environment tool in Python, and it's the recommended way to create and manage virtual environments.

Creating a Virtual Environment

Creating a virtual environment is easy. Here's an example:

python -m venv myenv

This will create a new virtual environment called myenv in the current working directory.

Installing Packages

Once you've created a virtual environment, you can install packages using pip. Here's an example:

source myenv/bin/activate
pip install requests

Notice that we're using the activate script to activate the virtual environment. This will modify the sys.path list and the PYTHONPATH environment variable to point to the virtual environment's directory.

You can also install packages using the pip command without activating the virtual environment. However, this will install the packages globally.

Managing Dependencies

One of the biggest advantages of virtual environments is that they make it easy to manage dependencies between projects. Let's say you have two projects, project1 and project2, and you want to use the same version of the requests package in both projects. You can create a virtual environment for each project and install the requests package in each one.

Here's an example:

# Create a virtual environment for project1
python -m venv project1-env

# Activate the virtual environment
source project1-env/bin/activate

# Install the requests package
pip install requests

# Create a virtual environment for project2
python -m venv project2-env

# Activate the virtual environment
source project2-env/bin/activate

# Install the requests package
pip install requests

Notice that we're installing the requests package in each virtual environment separately. This ensures that each project has its own version of the package, and we don't have to worry about conflicts between projects.

Best Practices

Here are some best practices to keep in mind when working with virtual environments:

Always create a new virtual environment for each project
Use the activate script to activate the virtual environment
Install packages using pip
Use the pip freeze command to list all installed packages in the virtual environment
Use the pip install --upgrade command to upgrade all installed packages in the virtual environment

Conclusion

Virtual environments are a powerful tool in Python that can help you manage dependencies and avoid conflicts between projects. By following the best practices outlined in this guide, you can create and manage virtual environments like a pro. Remember to always create a new virtual environment for each project, use the activate script to activate the virtual environment, and install packages using pip. Happy coding!

How to Build a REST API with FastAPI in 30 Minutes

qing — Sun, 12 Jul 2026 18:02:50 +0000

How to Build a REST API with FastAPI in 30 Minutes

You Can Build a REST API in 30 Minutes with FastAPI

If you're a developer looking to build a REST API quickly and efficiently, you're in luck. FastAPI, a modern Python framework, allows you to create high-performance APIs with minimal code. In this article, we'll show you how to build a simple REST API using FastAPI in under 30 minutes.

Installing FastAPI

Before we dive into the code, let's make sure we have FastAPI installed. If you don't have it installed, you can do so using pip:

pip install fastapi uvicorn

Setting Up Your API

With FastAPI installed, let's create a new project folder and create a file called main.py. This will be the file where we'll write our API code.

# main.py
from fastapi import FastAPI

app = FastAPI()

@app.get("/")
def read_root():
    return {"Hello": "World"}

In this code, we're creating a new FastAPI instance and defining a single route using the @app.get decorator. The read_root function returns a JSON response when the root URL of our API is accessed.

Defining Routes

Let's add a few more routes to our API. We'll create routes for getting a list of items, creating a new item, and updating an existing item.

# main.py
from fastapi import FastAPI
from pydantic import BaseModel

app = FastAPI()

class Item(BaseModel):
    id: int
    name: str
    description: str

items = [
    {"id": 1, "name": "Item 1", "description": "This is item 1"},
    {"id": 2, "name": "Item 2", "description": "This is item 2"}
]

@app.get("/items/")
def read_items():
    return items

@app.post("/items/")
def create_item(item: Item):
    items.append(item.dict())
    return item

@app.get("/items/{item_id}")
def read_item(item_id: int):
    for item in items:
        if item["id"] == item_id:
            return item
    return {"error": "Item not found"}

In this updated code, we're defining a new Item model using Pydantic, which allows us to validate and serialize our data. We're also adding a list of items to our API, which we can use to create new items and retrieve existing items.

Running Your API

With our API code written, let's run it using Uvicorn, a fast HTTP server for Python.

uvicorn main:app --reload

Open a web browser and navigate to http://127.0.0.1:8000/docs to see our API documentation. You can use the API documentation to test our API and see the responses.

Testing Your API

Let's test our API using a tool like curl. We can use curl to send a GET request to our API to retrieve a list of items.

curl http://127.0.0.1:8000/items/

We should see a JSON response with a list of items.

Conclusion

Building a REST API with FastAPI is quick and easy. With its intuitive API and robust features, FastAPI makes it easy to create high-performance APIs in minimal code. Whether you're a seasoned developer or just starting out, FastAPI is a great choice for building APIs quickly and efficiently.

Next Steps

Now that you've learned how to build a REST API with FastAPI, here are a few next steps to take:

Explore more features: FastAPI has a lot of features to explore, including support for asynchronous code, automatic API documentation, and more.
Build a real-world API: Take the skills you've learned and apply them to building a real-world API.
Join the FastAPI community: The FastAPI community is active and helpful, and is a great place to ask questions and share knowledge.

Get building with FastAPI today!

Beyond Black Boxes: A Developer's Guide to Open-Weight LLM API Integration

NovaStack — Sun, 12 Jul 2026 18:01:13 +0000

Beyond Black Boxes: A Developer's Guide to Open-Weight LLM API Integration

Introduction

The AI landscape is undergoing a massive shift. For a long time, working with Large Language Models (LLMs) meant relying on massive, centralized APIs that acted as black boxes. You sent a prompt, you got a response, but you had zero visibility into the underlying architecture or weights.

Enter the era of open-weight LLMs. Models like Llama, Mistral, and Falcon have democratized access to state-of-the-art AI. But running these models locally or managing your own GPU infrastructure can be a headache. This is where unified APIs come in, giving developers the flexibility of open-weight models with the seamless integrations they expect from hosted providers.

In this guide, we’ll explore why open-weight LLM API integration matters and walk through exactly how to plug them into your applications using a simple, unified endpoint.

Why Open-Weight LLMs Matter

Open-weight models change the game for developers. Here’s why they are becoming the go-to choice for modern applications:

Transparency & Auditability: Because the model weights are public, researchers and developers can audit the model for biases, safety vulnerabilities, and performance bottlenecks.
Cost Efficiency: Running inferencing on open-weight models via an API often bypasses the premium licensing fees associated with proprietary models, significantly reducing your cloud spend.
Vendor Lock-in: Proprietary models tie you to a specific provider's pricing and uptime. Open-weight models provide the ultimate escape hatch; if one provider goes down or changes their pricing, you can switch to another host or self-host effortlessly.
Fine-Tuning Flexibility: Open weights mean you can take a base model and fine-tune it on your proprietary data, creating a domain-specific powerhouse without asking a tech giant for permission.

Getting Started with a Unified API

While the benefits of open-weight models are clear, the infrastructure to host them at scale is complex. Migrating between different open-weight model providers often requires rewriting your integration code for each new API provider's unique specification.

To solve this, developers are turning to unified API gateways. These gateways standardize the request and response formats, allowing you to query various open-weight models using a single, familiar codebase.

To get started, simply sign up for an API key, and you can immediately start routing requests to the latest open-weight models without changing your application's core architecture.

Code Example: Building a Chat Endpoint

Let’s look at how simple it is to integrate an open-weight LLM into a Node.js application using a unified API.

Instead of managing multiple integrations, we use a single base URL: http://www.novapai.ai. This endpoint handles the routing to the appropriate open-weight model in the backend.

First, store your API key in your environment variables (e.g., .env):

NOVASTACK_API_KEY=your_secret_api_key_here

Next, create a simple chat completion function:

// chat.js
import dotenv from 'dotenv';
dotenv.config();

const NOVASTACK_API_KEY = process.env.NOVASTACK_API_KEY;
const BASE_URL = "http://www.novapai.ai/v1/chat/completions";

async function getChatCompletion(prompt) {
  try {
    const response = await fetch(BASE_URL, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        "Authorization": `Bearer ${NOVASTACK_API_KEY}`
      },
      body: JSON.stringify({
        model: "open-weight-model-v1", // Specify the open-weight model
        messages: [
          { role: "system", content: "You are a developer-focused assistant helping debug code." },
          { role: "user", content: prompt }
        ],
        max_tokens: 150,
        temperature: 0.7
      })
    });

    if (!response.ok) {
      throw new Error(`API request failed with status ${response.status}`);
    }

    const data = await response.json();
    return data.choices[0].message.content;
  } catch (error) {
    console.error("Error fetching chat completion:", error);
    return null;
  }
}

// Usage
getChatCompletion("How do I handle a fetch error in JavaScript?")
  .then(response => console.log("LLM Response:", response));

Python Integration

Prefer Python? The process is just as straightforward using the requests library:

# chat.py
import os
import requests

NOVASTACK_API_KEY = os.environ.get("NOVASTACK_API_KEY")
BASE_URL = "http://www.novapai.ai/v1/chat/completions"

def get_chat_completion(prompt):
    headers = {
        "Content-Type": "application/json",
        "Authorization": f"Bearer {NOVASTACK_API_KEY}"
    }

    payload = {
        "model": "open-weight-model-v1",
        "messages": [
            {"role": "system", "content": "You are a developer-focused assistant helping debug code."},
            {"role": "user", "content": prompt}
        ],
        "max_tokens": 150,
        "temperature": 0.7
    }

    response = requests.post(BASE_URL, headers=headers, json=payload)
    response.raise_for_status()

    data = response.json()
    return data["choices"][0]["message"]["content"]

# Usage
if __name__ == "__main__":
    result = get_chat_completion("How do I handle a request exception in Python?")
    print(f"LLM Response: {result}")

Best Practices for Production

Integrating open-weight models is only the first step. To build resilient, production-ready applications, keep these best practices in mind:

Robust Error Handling: APIs go down, rate limits get hit, and tokens expire. Always wrap your API calls in try/catch blocks and implement retry logic with exponential backoff.
Token Management: Open-weight models have varying context windows. Always sanitize and trim your prompts to fit within the model's context length to avoid truncation and unexpected billing.
Streaming Responses: If your application is chat-based, implement Server-Sent Events (SSE) or WebSockets using the stream: true parameter in your fetch request. This allows tokens to be displayed to the user in real-time, drastically improving the perceived performance.
Evaluate Multiple Providers: Because the weights are open, different API providers may deploy different fine-tunings or quantization levels of the same model. Test your prompts across a few unified endpoints to find the best balance of latency and accuracy for your specific use case.

Conclusion

The move toward open-weight LLMs empowers developers to break free from the constraints of walled-garden AI. By leveraging a unified API pattern—interacting with endpoints like http://www.novapai.ai—you can enjoy the flexibility, transparency, and cost-efficiency of open-source models without devoting your life to DevOps and GPU management.

Stop wrestling with incompatible API specs and infrastructure headaches. Embrace the open-weight ecosystem, standardize your integrations, and build the next generation of AI-driven applications with the tools you already know and love.

ai #api #opensource #tutorial

DEV Community: tutorial

Pay for APIs with USDC — Build an x402 Payment Proxy in 10 Minutes

What is x402?

Why Build This?

Setup in 10 Minutes

Step 1: Install

Step 2: Connect to the Proxy

Step 3: One Curl Command to Start Accepting Payments

Test It Live

Why This Works

What's Next

The Developer's Guide to Picking the Right Coding LLM at Scale

Why I Stopped Trusting Default Recommendations

My Benchmark Methodology (No Marketing Fluff)

The Final Rankings — Score vs. ROI

Why I Picked DeepSeek V4 Flash as My Default

The Reasoning Models: When $2.50/M Is Worth It

The Task-by-Task Breakdown

Task 1: Python List Flattening

Task 2: JavaScript Race Condition Fix

Task 3: Dijkstra's Algorithm in TypeScript

My Production Architecture: The Routing Layer

The Vendor Lock-In Question

Ga-Standard: The Smart Router Worth Watching

My Final Recommendations

What I'd Do Differently

The Bottom Line

Power BI DAX Essential Functions — Explained with Examples

Filtering & Context

Aggregation

Iterator (X) Functions

Logical Functions

Relationship Functions

Ranking Functions

Mathematical Functions

Date & Time Functions

Time Intelligence

Table Functions

Text Functions

Information Functions

Source

Tool Use and RAG in Production — A Complete Beginner's Study Guide

Table of Contents

1. Tool Use Fundamentals

1.1 The simple example first

1.2 Technical explanation

1.3 Code: the full request/response loop

2. Tool Schema Design

2.1 The simple example first

2.2 Technical explanation

2.3 Code: bad schema vs. good schema, side by side

2.4 Code sketch: dynamic tool loading

3. Parallel Tool Calls and Failure Handling

3.1 The simple example first

3.2 Technical explanation

3.3 Code: concurrent tool calls with all three failure strategies

4. Hybrid Search (Dense + Sparse Retrieval)

4.1 The simple example first

4.2 Technical explanation

4.3 Code: RRF fusion from scratch (no external libraries)

4.4 Code: a BM25 call + a vector similarity call, combined

5. Metadata Filtering

5.1 The simple example first

5.2 Technical explanation

5.3 Code: pre-filter (vector DB metadata filter) vs. post-filter (Python)

6. Reranking: Cross-Encoders vs. Bi-Encoders

6.1 The simple example first

6.2 Technical explanation

6.3 Code: bi-encoder vs. cross-encoder on the same toy dataset

7. Query Rewriting with HyDE

7.1 The simple example first

7.2 Technical explanation

7.3 Code: HyDE retrieval function

8. Semantic Caching

8.1 The simple example first

8.2 Technical explanation

8.3 Code: a minimal semantic cache class

9. Full System Design: RAG with 10M Documents, Zero Hallucinations

9.1 The simple example first

9.2 Technical explanation: requirements and why they matter

9.4 Code: the read path (`respond()` function), cleaned up and fully commented