DEV Community: Thousand Miles AI

How LLMs Actually Generate Text — Temperature, Top-K, Top-P, and the Dice Rolls You Never See

Thousand Miles AI — Fri, 06 Mar 2026 05:46:02 +0000

You set temperature to 0.7 because a tutorial told you to. But do you know what that actually does? Under the hood of every LLM response is a probability game — here's how the dice are loaded.

How LLMs Actually Generate Text — Temperature, Top-K, Top-P, and the Dice Rolls You Never See

Every token an LLM outputs is a gamble. Understanding how that gamble works changes how you use these models forever.

The Same Prompt, Three Different Answers

Try this experiment. Open any LLM — ChatGPT, Claude, Gemini, whatever you have access to. Ask it: "Write a one-sentence product description for a coffee mug." Hit send. Copy the result. Now ask the exact same question again. And again.

Three attempts. Three different sentences. Maybe slightly different, maybe wildly different. But almost certainly not identical.

Why? You gave it the exact same input. The model's weights didn't change between requests. The system prompt is the same. So where does the randomness come from?

It comes from the sampling step — the moment after the model calculates probabilities for every possible next word, and before it actually picks one. That choice — how the model selects from thousands of candidates — is controlled by parameters you've probably seen but maybe never understood: temperature, top-K, top-P.

These aren't minor settings. They fundamentally change the model's behavior. Get them wrong, and your creative writing tool sounds robotic. Or your code assistant hallucinates syntax that doesn't exist. Or your customer support bot gives a different answer to the same question every time.

Why Should You Care?

If you're building anything with an LLM — even just making API calls — you're setting these parameters, whether you know it or not. Every API has defaults. Every playground has sliders. And most developers just leave them alone or copy values from tutorials without understanding what they do.

Understanding sampling isn't academic — it's one of the highest-leverage ways to improve LLM output quality without changing a single word of your prompt. It also shows up in interviews constantly. "Explain how temperature works" is practically a warmup question at any AI-focused company.

Let Me Back Up — How an LLM Picks the Next Word

Here's what happens every time an LLM generates a single token:

The model processes your input and produces a set of logits — raw scores for every token in its vocabulary (typically 30,000–100,000+ tokens).
Those logits go through a softmax function, which converts them into probabilities that sum to 1.
A sampling strategy picks one token from that probability distribution.
That token gets appended to the output, and the whole process repeats for the next token.

The model generates text one token at a time, left to right. It doesn't plan ahead. It doesn't have a draft that it edits. Every single token is a fresh probabilistic choice based on everything that came before it.

The generation loop: predict probabilities for all tokens, sample one, append, repeat. The sampling strategy is where the magic (and danger) happens.

The Sampling Strategies — One by One

Greedy Decoding: Always Pick the Winner

The simplest strategy. At every step, pick the token with the highest probability. No randomness, no dice rolling. If "the" has probability 0.35 and "a" has 0.20, you always pick "the."

Sounds sensible, right? But greedy decoding has a nasty problem: it's boring. It tends to produce repetitive, predictable text. It gets stuck in loops. It picks the "safe" word every time, and the result reads like it was written by someone who's afraid to take any creative risk.

Greedy decoding is fine for tasks where you want the single most likely answer — like classification or extraction. For anything generative, it's almost never what you want.

Temperature: Turning Up the Creativity Dial

Temperature is the parameter everyone knows and almost nobody understands precisely. Here's what it actually does.

Before the softmax function converts logits to probabilities, temperature divides the logits by a number. That's it. That's the whole mechanism.

But the effect is dramatic:

Temperature = 1.0 — No change. The probabilities are whatever the model naturally produces.

Temperature < 1.0 (say, 0.3) — The logits get divided by a small number, which amplifies the differences between them. High-probability tokens become even more probable. Low-probability tokens become nearly impossible. The distribution gets "peaky" — the model becomes more confident, more predictable, more conservative.

Temperature > 1.0 (say, 1.5) — The logits get divided by a large number, which flattens the differences. Every token becomes more equally likely. The distribution spreads out — the model becomes more random, more creative, more surprising. Also more likely to say something unhinged.

Think of temperature like a volume knob for randomness. Turn it down for math homework. Turn it up for poetry. Turn it all the way down (temperature = 0) and you get greedy decoding — pure determinism.

Top-K Sampling: Only Consider the Top Candidates

Top-K is a filter. Before sampling, it looks at all 50,000+ tokens in the vocabulary, keeps only the K most probable ones, and throws the rest away. The probability mass gets redistributed among the survivors.

Set K = 50, and the model can only choose from its top 50 candidates. Set K = 5, and it's stuck with the top 5. Set K = 1, and you're back to greedy decoding.

The problem with top-K? The number K is fixed, regardless of context. Sometimes the model is very confident — 3 tokens account for 95% of the probability, and everything else is noise. A K of 50 would include 47 tokens that have almost zero chance of being right. Other times the model is uncertain — 200 tokens each have a small but meaningful probability. A K of 50 would cut off potentially good options.

Top-K doesn't adapt to the shape of the distribution. It's blunt.

Top-P (Nucleus Sampling): The Smart Filter

Top-P, also called nucleus sampling, is the clever answer to top-K's rigidity. Instead of keeping a fixed number of tokens, it keeps the smallest set of tokens whose combined probability exceeds a threshold P.

Set P = 0.9, and the model keeps adding tokens (from most to least probable) until their probabilities sum to 0.9. If the model is confident, that might be only 3 tokens. If the model is uncertain, it might be 200.

The beauty is that top-P adapts to context. When the next word is obvious ("The Eiffel Tower is in __"), it narrows down to very few candidates. When the next word could genuinely go many ways ("She felt __"), it keeps a wider pool.

This is why top-P has become the default sampling strategy in most production systems. It's more robust across different situations than top-K.

Top-K keeps a fixed number regardless of confidence. Top-P adapts — tight when confident, wide when uncertain.

Min-P: The 2026 Newcomer

There's a newer approach that's gaining traction, especially in open-source communities. Min-P sets a threshold relative to the most probable token. If the top token has probability 0.8 and min-P is 0.1, any token with probability below 0.08 (10% of 0.8) gets cut.

The elegance is that it scales with the model's own confidence. When the model is very sure (top token at 0.95), the threshold is high and very few alternatives survive. When the model is less sure (top token at 0.2), the threshold drops and more tokens stay in the pool.

As of early 2026, the combination of temperature + min-P is what many open-source LLM users have converged on as the most practical setup.

Practical Guide: What Settings for What Task

Here's a cheat sheet based on how these strategies interact:

Code generation, factual Q&A, data extraction: Temperature 0–0.3, top-P 0.9. You want determinism and accuracy. The model should pick the most likely token almost every time.

General chatbot, customer support: Temperature 0.5–0.7, top-P 0.9. A balance of reliability and natural-sounding language. Not robotic, not chaotic.

Creative writing, brainstorming, poetry: Temperature 0.8–1.2, top-P 0.95. Give the model room to explore. Higher temperature means more surprising word choices.

Never go above 1.5 for temperature unless you're doing it for fun. At that point, the probability distribution is so flat that the model starts producing incoherent output — like a writer who's had too much coffee and is just free-associating.

Mistakes That Bite — Common Misunderstandings

"Temperature controls how smart the model is." No. It controls the randomness of token selection. A low temperature doesn't make the model think harder — it makes it pick the highest-probability token more consistently. If the model's probabilities are wrong, low temperature just makes it confidently wrong.

"I should always use top-K AND top-P together." You can, but be careful. If you set K=50 and P=0.9, the effective filter is whichever is more restrictive. Often one overrides the other, and the second parameter does nothing. Pick one or understand how they interact in your specific framework.

"Temperature 0 means the same output every time." Almost. It means greedy decoding — always picking the highest-probability token. But some implementations have floating-point tie-breaking that can occasionally vary. For true determinism, also set a fixed random seed if the API supports it.

Now Go Break Something — Where to Go from Here

The best way to internalize these concepts is to play with them:

Use the OpenAI or Anthropic playground — they have real-time sliders for temperature, top-P, and top-K. Ask the same question at different settings and watch how the output changes.
Try the Hugging Face text generation playground — it shows the token probabilities alongside the generated text, so you can literally see the dice being rolled.
Search for "LLM sampling parameters interactive demo" — several blog posts have visual explainers that let you see how temperature reshapes the probability distribution.
Read the Hugging Face blog post "Decoding Strategies in Large Language Models" — it covers everything from greedy search to min-P with code examples.
For open-source users: Experiment with llama.cpp's sampler chain — it lets you compose multiple sampling strategies in sequence and see how each one transforms the distribution.

Next time you set temperature to 0.7 and top-P to 0.95, you'll know exactly what's happening: the model calculates probabilities for 50,000 tokens, temperature sharpens the distribution slightly, top-P keeps only the tokens that matter, and one gets picked. Every word you read from an LLM went through this gauntlet. The same prompt, the same model, but different dice rolls — and that's why you get a different coffee mug description every time.

Author: Shibin

The LLM Interview Cheat Sheet — 10 Questions That Actually Come Up

Thousand Miles AI — Fri, 06 Mar 2026 05:44:59 +0000

You've used ChatGPT, built a RAG pipeline, maybe even fine-tuned a model. But can you explain how attention actually works when the interviewer asks? Here are 10 LLM questions that keep showing up in interviews — with answers that actually make sense.

It's 10 PM the night before your Google / Meta / OpenAI LLM engineer interview. You're scrolling through your notes on transformers, and your mind goes blank when you try to explain self-attention out loud. You panic. You Google "explain attention mechanisms" and spend the next hour reading academic papers that feel written in a different language.

By midnight, you're convinced you don't know anything.

Here's the truth: you probably know more than you think. You've fine-tuned models, built RAG pipelines, maybe even experimented with prompt engineering. But when an interviewer asks "How does self-attention work?" or "When would you use fine-tuning vs RAG?", panic takes over and you blank out.

This post is your cheat sheet. Not the academic definitions. The answers that actually work in an interview — clear, concise, and confident.

Why You Should Care

LLM roles are exploding right now. Google, Meta, OpenAI, Anthropic, Microsoft — they're all hiring ML engineers who can talk intelligently about transformers, RAG, fine-tuning, and hallucination. These aren't niche roles anymore. They're the growth area in tech.

The gatekeepers for these roles are these 10 questions (or variations of them). They appear across companies because they separate people who understand LLMs from people who just know how to use them.

The good news: these questions have predictable answers. You just need to know how to explain them.

The 10 Questions (+ Answers You Can Deliver)

1. Explain self-attention. Why can't you just use RNNs?

Why they ask: This is the foundation of everything. If you can't explain this clearly, everything else falls apart.

The answer:
Self-attention lets a token look at every other token in the sequence at once and assign weights to determine which ones matter. It answers: "Given this token, which other tokens should I pay attention to?"

Here's the concrete difference:

RNN (old way): Processes tokens one at a time, left to right. Token at position 10 struggles to "remember" token at position 1 because information has to flow through 9 steps. Long dependencies get lost.
Self-attention (new way): Token at position 10 directly computes its similarity to all other tokens (positions 1–9) and decides their importance instantly. No information decay.

The formula you don't need to memorize, but should understand:

Attention(Q, K, V) = softmax(Q * K / sqrt(d)) * V

Translation: Take your query (Q), multiply by all keys (K), normalize with softmax so weights sum to 1, then multiply by the actual values (V).

The gotcha: Interviewers might ask "What's the computational cost?" Answer: O(n²) where n is sequence length. That's why long context windows are expensive. That's also why companies invest in optimized attention (multi-query attention, flash attention).

2. What is positional encoding and why do we need it?

Why they ask: Transformers are permutation-invariant (word order doesn't matter by default). They want to know if you understand why that's broken and how we fix it.

The answer:
Self-attention doesn't inherently know position. If you feed "dog bit man" or "man bit dog", the attention mechanism computes the same weights. The model needs to know which word is first, second, third.

Positional encoding adds information about position to each token's embedding. The most common method (from the original paper) uses sin/cos waves at different frequencies:

Low frequencies encode large-scale positions (is this early or late in the sequence?)
High frequencies encode local positions (is this token next to another one?)

This way, the model can learn relationships like "noun at position 2, verb at position 4" instead of just "noun, verb".

The gotcha: There's no single best positional encoding. Some models use learned positional embeddings. Others use relative position bias. What matters is that you know the problem exists.

3. Self-attention, multi-head attention — what's the difference?

Why they ask: This trips up a lot of candidates. They use the term "attention head" without understanding what it does.

The answer:
Self-attention is the basic mechanism (Q, K, V multiply and softmax).

Multi-head attention is running the same self-attention operation multiple times in parallel, each with different weight matrices, then combining the results.

Why? Because different "heads" can learn different patterns:

One head might learn to focus on nearby words (local grammar)
Another head might learn to focus on distant words (long-range references)
A third head might learn to focus on certain semantic relationships

Think of it like having 8 different "experts" all looking at the same input but with different lenses.

Formula-wise: Instead of one attention output, you get multiple outputs and concatenate them.

The gotcha: Having 8 heads doesn't mean 8× the understanding. Empirically, 8–16 heads work well. More isn't always better (there are diminishing returns).

4. Explain the transformer architecture in 30 seconds.

Why they ask: They want to know if you can break down complexity. If you ramble for 5 minutes, they think you don't understand the core.

The answer (say this fast):
Transformer has two parts: encoder and decoder.

Encoder: Takes input text, runs it through self-attention (to let tokens attend to each other), then through a feed-forward network. Do this 12–24 times (stacking layers). Output: rich representation of the input.

Decoder: Takes target tokens, runs self-attention (but masked so it can't look ahead), then cross-attention (attends to encoder output), then feed-forward. Do this 12–24 times. Output: next token prediction.

In one sentence: "Stack self-attention and feed-forward layers, apply masking in the decoder, and train to predict the next token."

5. What is tokenization and why does it matter?

Why they ask: Tokenization is the first step. Get it wrong and everything downstream breaks. They want to know if you've thought about this.

The answer:
Tokenization converts raw text into tokens (usually subwords) that the model can process.

"Hello world" might become ["Hel", "lo", "world"] or ["Hello", "world"] depending on the tokenizer.

Why subwords instead of just words?

Rare words: If the tokenizer has never seen "pneumonia", breaking it into ["pneu", "monia"] lets the model handle it anyway
Efficiency: Fewer tokens = faster processing
Spelling variations: "color" and "colour" map to similar tokens

Two main approaches:

BPE (Byte Pair Encoding): Used by GPT. Learns common character pairs and merges them iteratively
WordPiece: Used by BERT. Similar idea but with a frequency-based approach

The gotcha: Different models use different tokenizers. GPT-4 uses a different tokenizer than GPT-3. This matters for token counting, context window size, and fine-tuning.

6. Explain the difference between fine-tuning and RAG. When would you use each?

Why they ask: This separates people building LLM products from people who understand the tradeoffs. It's a systems thinking question.

The answer:

Aspect	Fine-tuning	RAG
What it does	Adjusts model weights on your task-specific data	Retrieves relevant docs, adds them to prompt before generation
Cost	Expensive (GPU hours, time)	Cheap (just needs retrieval + inference)
Speed	Slow to deploy	Fast to iterate
Knowledge cutoff	Can be months/years old (trained on historical data)	Can include live, up-to-date information
When to use	Specific writing style, domain-specific reasoning, behavior you can't prompt into the model	Factual Q&A, company docs, changing information

The real answer: Most of the time, start with RAG. It's faster to build and easier to maintain. Use fine-tuning only when:

You have lots of labeled examples (1000+)
You need consistent style/format
RAG isn't getting you there
You have the infrastructure to maintain a custom model

Example: Customer support chatbot? RAG + the company's knowledge base. Custom code generation for your codebase? Fine-tuning might be worth it.

7. What causes hallucination in LLMs and how do you prevent it?

Why they ask: Hallucination is the biggest issue in production LLM systems. They want to know if you've dealt with it.

The answer:

What is hallucination? The model generates confident, fluent text that's completely false. Not random gibberish — plausible-sounding facts that are wrong.

Why it happens:

The model predicts the next most-likely token based on pattern matching, not factual knowledge
It hasn't learned the boundary between "I know this" and "I'm guessing"
It's trained to be coherent, not accurate

How to prevent it (in order of effectiveness):

RAG (best solution): Give the model a document to read from. Now it can only hallucinate based on what's in that document. Most controllable.
Prompt engineering: Explicit instructions like "Only answer based on the provided context" or "If unsure, say 'I don't know'" help a bit. But models still hallucinate.
Fine-tuning on high-quality data: Train the model on examples where it's penalized for hallucinating. Helps but doesn't fully solve it.
Fact-checking layer: After generation, run the output through a separate fact-checker (another model or rule-based system).
Temperature control: Lower temperature makes the model more confident in likely tokens, reduces randomness. But doesn't fix hallucination.

The honest answer: You can't eliminate hallucination. You can reduce it. RAG is your best bet.

8. How would you evaluate an LLM's quality?

Why they ask: Generating text is easy. Knowing if it's good is hard. They want to know if you've thought about measurement.

The answer:
Depends on the task. There's no one metric.

Automatic metrics (cheap, noisy):

BLEU, ROUGE: Compare generated text to reference text word-by-word. Works for translation, summarization. Penalizes paraphrasing. Bad for open-ended tasks.
BERTScore: Uses embeddings instead of exact word match. More forgiving. Better than BLEU.
Exact Match (EM), F1: For QA. Did the model extract the right answer?

Manual evaluation (expensive, signal-rich):

Human raters: Have people score outputs (1–5) on relevance, accuracy, tone. Gold standard. Requires budget.
Rubric-based: Define criteria (factuality, clarity, completeness) and score against them.

LLM-as-a-Judge (emerging, controversial):

Use a strong LLM (GPT-4) to score outputs from a weaker LLM. Fast and surprisingly good, but can be circular (errors compound).

Business metrics:

For a chatbot: user satisfaction, conversation length, return rate
For a code generator: does generated code compile? Does it pass tests?

The honest answer: Use multiple signals. No single metric tells the full story.

9. Explain what's happening in a forward pass through a transformer.

Why they ask: They want to verify you can trace through the actual computation. Not just regurgitate definitions.

The answer:

Step by step:

Tokenization: "Hello world" → 101, 7592, 2088
Embedding: Each token ID maps to a d-dimensional vector (e.g., 768D for BERT)
Positional encoding: Add sin/cos vectors so the model knows position
Transformer block: Run through self-attention, feed-forward, repeat 12+ times. Each layer transforms the embeddings, extracting deeper meaning
Output layer: Linear layer that converts final embeddings to logits (scores) for each possible next token
Softmax: Convert logits to probabilities summing to 1
Sampling: Pick the next token (or the highest probability one)
Repeat: Feed the new token back in, keep going

The gotcha: During inference, you don't recompute attention over all previous tokens every time (too expensive). You use KV caching: store the keys and values from previous tokens, reuse them, only compute for new tokens.

10. What's the difference between base models and instruction-tuned models? Why do we need both?

Why they ask: This is about understanding the training pipeline and product strategy. It separates engineers from researchers.

The answer:

Base models (like GPT-3, LLaMA):

Trained on next-token prediction on huge internet text
Excellent at patterns and language
Terrible at following instructions
If you ask "Tell me a joke", it will continue text in a way that follows common patterns, not necessarily tell a joke
Useful for: creative writing, text completion, in-context learning

Instruction-tuned models (like ChatGPT, Llama-2-Chat):

Take a base model
Fine-tune it on (instruction, response) pairs where responses are aligned with what users want
Also fine-tune with RLHF (Reinforcement Learning from Human Feedback) to penalize bad outputs
Follows instructions reliably
Useful for: chatbots, Q&A, customer support

Why both exist:

Base models are research tools. They're the raw material.
Instruction-tuned models are products. They're what users interact with.
Sometimes you want a base model (if you're doing research or building something unusual). Usually you want instruction-tuned (if you're shipping to users).

The gotcha: Fine-tuning an instruction-tuned model on new data can degrade instruction-following. This is catastrophic forgetting. You need to be careful about the training setup.

Common Gotchas (Things Candidates Mess Up)

Confusing attention with RNNs: Attention is not sequential. RNNs are sequential. Don't say "attention is better because it's faster at each step" — say "it's faster overall because steps are parallelizable".
Overstating transformer improvements: Transformers are great at long context, but they have O(n²) memory. This is a real limitation. Don't pretend it doesn't exist.
Assuming fine-tuning is always the answer: Most people reach for fine-tuning too early. RAG, prompting, and in-context learning go further than most engineers think.
Saying "more parameters = better": Scaling helps, but data quality and training setup matter just as much. A 7B model trained right beats a 70B model trained poorly.
Forgetting the practical constraints: Interviewers care about inference cost, latency, and inference cost. Academic perfection doesn't matter if you can't serve it.
Not understanding your own tools: If you've used OpenAI API, know its pricing, latency, rate limits. If you've fine-tuned on Hugging Face, know how long it takes and what it costs. Specifics matter.

What to Do Next

Practice explaining these answers out loud. Not reading — speaking. Your brain works differently. You'll stumble on things you thought you understood.
Build something. Try a RAG system, fine-tune a model on your own data, or build a chatbot. Theory is fine, but interviewers test your judgment. You get that from building.
Read the original papers lightly. Not cover-to-cover. Read "Attention is All You Need" (Vaswani et al., 2017) for context. Skim the abstract and architecture section. You don't need to memorize it.
Know your specific tech stack. If you're interviewing at a company, know what models they use. Google? PaLM and Gemini. Meta? LLaMA. OpenAI? GPT-4. Anthropic? Claude. Know the positioning. It shows you've done your homework.
Practice system design questions. "Design a chatbot for a healthcare provider" or "Design a code generation service." These combine everything. Most interviews include one.

The Night Before

You're going to be nervous. That's normal. Everyone is.

The difference between people who pass and people who don't isn't knowledge — it's clarity. You probably know 80% of what you need. You just need to deliver it with confidence.

Before bed, do this:

Read through your answers once (not for hours — 20 minutes max)
Do a practice explanation out loud for each question
Go to sleep knowing that you've prepped well

During the interview:

If they ask something you don't know, say "I don't know that specific detail, but here's how I'd think about it." Then reason through it. Reasoning is more valuable than memorization.
If you blank out on a question, pause for 5 seconds. Think. Then answer. Silence is okay. Rambling is bad.

You've got this. Good luck.

Sources:

Author: thousandmiles-ai-admin

How AI Agents Actually Execute Multi-Step Tasks — The Orchestration Nobody Talks About

Thousand Miles AI — Fri, 06 Mar 2026 05:43:56 +0000

You asked the AI to 'book a flight and update the spreadsheet.' It did both. But how? A deep dive into the reasoning loop, tool calling, and orchestration patterns that make AI agents actually work.

How AI Agents Actually Execute Multi-Step Tasks — The Orchestration Nobody Talks About

An LLM can write poetry and explain quantum physics. But ask it to "check the database, find stale records, and send a Slack alert" — and suddenly it needs an entire architecture to pull it off.

The "Just Do It" Illusion

You're watching a demo. Someone types into a chat: "Find all overdue invoices in our system, calculate the total amount, and draft an email to the finance team with a summary." The AI assistant thinks for a moment, then — like magic — it queries the database, crunches the numbers, writes a professional email, and asks for confirmation before sending.

It looks seamless. Like the AI just... understood and did everything. But behind that smooth demo is something much more interesting: a loop. The AI didn't do all of that in one shot. It thought about what to do first, executed one step, looked at the result, thought again, executed the next step, and kept going until the job was done.

That loop — the reasoning-action-observation cycle — is the beating heart of every AI agent. And understanding it is the difference between building chatbots that answer questions and building agents that actually get things done.

Why Should You Care?

If you're building anything with LLMs beyond a simple Q&A bot — a coding assistant, an automated workflow, an internal ops tool — you're building an agent, whether you call it that or not. And every AI-focused company, from startups to the big labs, is hiring people who understand how agents work under the hood.

More practically: the agent architecture you choose determines whether your system is reliable or a house of cards. The "just let the LLM figure it out" approach works in demos. In production, it falls apart spectacularly.

Let Me Back Up — What Even Is an AI Agent?

Let's get precise. An AI agent is an LLM-powered system that can take actions in the real world — not just generate text, but call APIs, query databases, read files, send messages, execute code. It does this autonomously, deciding on its own what steps to take to achieve a goal.

The key word is "autonomously." A regular LLM call is like asking someone a question — you get an answer back. An agent is like giving someone a task — they figure out the steps, do the work, and come back with results.

But here's the thing: LLMs don't inherently know how to plan and execute multi-step tasks. They're trained to predict the next token. The agent behavior comes from the architecture wrapped around the LLM — the loop, the tools, the memory, the orchestration logic. The LLM is the brain. Everything else is the body.

Okay, But How Does It Actually Work? — The ReAct Loop

The most foundational pattern in agent design is called ReAct — short for "Reasoning and Acting." It was introduced in a 2023 research paper, and by 2026 it's become the default mental model for how agents operate.

Here's the core idea: instead of asking the LLM to produce a final answer in one shot, you put it in a loop where it alternates between thinking and doing.

The ReAct loop: think, act, observe, repeat. Each cycle brings the agent closer to the goal.

Step by Step

Thought — The LLM generates internal reasoning. It looks at the goal, considers what information it has, and decides what to do next. This is essentially chain-of-thought reasoning, but directed toward action. Something like: "The user wants overdue invoices. I need to query the database first. I'll use the query_invoices tool with a filter for overdue status."

Action — Based on the thought, the LLM outputs a structured tool call. It's not free-form text — it's a specific function name with specific parameters, like query_invoices(status="overdue", limit=100). The orchestration layer parses this and executes it.

Observation — The tool runs, and its output gets fed back to the LLM as context. "Found 23 overdue invoices totaling $47,250." Now the LLM has new information it didn't have before.

Loop — The LLM sees the observation, generates a new thought ("Now I need to calculate the total and draft the email"), and takes the next action. This continues until the goal is met or the agent decides it needs human input.

The beauty of this pattern is that it's self-correcting. If a tool call fails, the LLM sees the error in the observation step and can try a different approach. If it gets unexpected data, it can reason about what went wrong. This feedback loop is what makes agents feel intelligent — they're not just following a script, they're adapting.

Why Not Just Plan Everything Upfront?

Fair question. Why not have the LLM create a full plan at the beginning and then execute it linearly? Some architectures do this — and it works for simple, predictable tasks. But for anything complex, upfront planning breaks down because the agent doesn't know what it'll discover along the way. Maybe the database query returns no results. Maybe the API is down. Maybe the data looks different than expected. The iterative loop handles uncertainty by making decisions one step at a time, with real information.

The Three Orchestration Architectures

Not all agents are built the same. As tasks get more complex, the simple single-loop pattern needs to evolve. Here are the three main architectures you'll see in production systems.

1. Single Agent Loop

This is the ReAct pattern we just described — one LLM handling everything end to end. It reads the goal, picks a tool, observes the result, and repeats.

Good for: Simple-to-moderate tasks with a clear sequence of steps. Think "search for X, summarize it, save to a file."

Breaks down when: The task requires expertise in multiple domains, or the number of available tools is so large that the LLM gets confused about which to use. When you give a single agent 50 tools, it starts picking the wrong ones — there's a real cognitive overload problem with tool selection.

2. Supervisor Pattern (Hierarchical)

A supervisor agent breaks the goal into sub-tasks and delegates each to a specialist agent. The supervisor doesn't do the work itself — it coordinates.

Think of it like a tech lead assigning tickets. The supervisor says: "Agent A, query the database for overdue invoices. Agent B, once A is done, calculate the total. Agent C, draft the email with the results."

Each worker agent runs its own ReAct loop with a narrower focus and fewer tools. The supervisor collects results and produces the final output.

Supervisor pattern: one coordinator, multiple specialist workers. Each worker has a focused role and limited tool set.

Good for: Complex tasks that need different types of expertise. One agent might be great with databases, another with writing, another with code.

Trade-off: More overhead. You're running multiple LLM calls, and the supervisor needs to correctly decompose the task. Bad decomposition means bad results.

3. Plan-Execute-Synthesize

This is the architecture that's gaining the most traction in 2026. It separates the agent into three distinct roles:

Planner — Looks at the goal and produces a structured plan. Just the plan — no execution. This forces the planning step to be explicit and reviewable.

Executor — Takes the plan and runs it step by step, calling tools and collecting results. The executor can only do what the plan authorizes. This makes the system predictable and auditable.

Synthesizer — Reads all the collected evidence (tool outputs, intermediate results) and composes the final answer. It never calls tools directly — it just works with the data.

Why this matters: By separating planning from execution from synthesis, you can enforce policies (the executor can't go rogue), audit every step (the plan is inspectable), and debug failures precisely (was the plan wrong? did a tool fail? did the synthesis miss something?).

Mistakes That Bite — Where Agent Architectures Go Wrong

"Give the agent all the tools and let it figure it out." This is the most common mistake. More tools does not mean more capability — it means more confusion. LLMs have a harder time choosing the right tool when the selection is large. Be surgical: give each agent only the tools it needs for its specific role.

"The LLM will handle error recovery." Sometimes. But LLMs can also get stuck in loops — calling the same failing tool over and over with slightly different parameters, burning tokens without making progress. Production agents need hard limits: maximum loop iterations, timeout policies, and escalation to a human when the agent is clearly stuck.

"We don't need a human in the loop." For low-stakes tasks like summarizing data, sure. But for anything that sends emails, modifies databases, or takes irreversible actions? You need a confirmation step. The best agent architectures have explicit "checkpoints" where the agent pauses and asks for human approval before proceeding with high-impact actions.

Now Go Break Something — Where to Go from Here

If you want to build your own agent and feel these patterns firsthand, here's a path:

Start with the ReAct pattern. Build a simple agent that has 2–3 tools (a web search tool, a calculator, and a file writer). Give it a goal that requires using all three. Watch how it reasons through the steps.
Try LangGraph — it lets you define agent workflows as graphs, which makes the orchestration patterns visual and easy to experiment with. The official docs have great quickstart tutorials.
Explore the OpenAI Agents SDK — it's lightweight and has built-in support for tool calling and MCP integration. Good for understanding the basics without framework overhead.
Read the original ReAct paper — search for "ReAct: Synergizing Reasoning and Acting in Language Models" by Yao et al. It's surprisingly readable for an academic paper, and understanding the origin helps you see why everything is built this way.
For the ambitious: Build a supervisor-worker system where a planner agent delegates to two specialist agents. Even a toy example with made-up tools will teach you more about orchestration challenges than any tutorial.

That seamless demo — where the AI queried the database, crunched numbers, and drafted an email — wasn't magic. It was a loop: think, act, observe, repeat. The LLM provided the reasoning. The orchestration provided the structure. And the tools provided the hands. Once you see the loop, every AI agent stops being a black box and starts being an engineering problem you can actually debug.

Author: thousandmiles-ai-admin

How to Evaluate LLM Outputs — Beyond 'Looks Good to Me'

Thousand Miles AI — Fri, 06 Mar 2026 05:42:53 +0000

Your RAG pipeline returns an answer. It sounds confident. But is it actually correct? Turns out 'vibes-based evaluation' doesn't scale. Learn the metrics and frameworks that actually tell you if your LLM is hallucinating, missing context, or nailing it.

The Classic Problem

You've built a RAG pipeline. Your knowledge base is solid. Your retriever works fine. You run a test query, and the LLM spits out an answer that sounds completely confident. Grammar? Perfect. Structure? Coherent. Tone? Professional.

You copy it to your Slack channel: "It works!"

But then someone asks a follow-up question, and the answer contradicts itself. Or they check a fact and it's subtly wrong. Or they ask "where did you get that?" and you realize the LLM just... made it up.

That's not an error in your pipeline. That's an error in how you evaluated it.

If you're relying on "it looks right to me," you're in dangerous territory. The problem scales immediately: when you have 100 queries, 1000 queries, or a production system running 24/7, you can't manually inspect every output.

You need metrics.

Why This Matters

Vibes-based evaluation breaks at scale. Human inspection is slow, inconsistent, and subjective. One person reads an answer and thinks "solid." Another reads the same answer and spots a hallucination. You're shipping an LLM system that nobody actually understands, and nobody can debug when it fails.

But here's the thing: traditional ML evaluation metrics don't work for language models. In classification, you have clear right/wrong answers. In RAG, there's no single "ground truth." The same query might have 10 correct answers depending on how you interpret it. And hallucinations are genuinely hard to spot automatically because the LLM is confident and grammatically flawless.

So we need new frameworks. We need to measure different dimensions separately, and we need tools that don't require a human to read every single output.

The Core Challenge: Why LLM Eval Is Different

Traditional ML evaluation assumes:

There's one right answer (binary classification, exact match, etc.)
Metrics are purely numerical (accuracy, precision, recall)
No middle ground between right and wrong

Language generation throws all of that out:

Multiple correct answers exist (paraphrasing, different phrasings, different correct facts)
Quality is multidimensional (you need to measure faithfulness and relevance, not just "accuracy")
Hallucinations look like correct answers—that's the whole problem

That's why LLM evaluation has evolved into a multi-metric framework where you evaluate different dimensions separately.

The RAGAS Framework: Your Evaluation Toolkit

RAGAS (Retrieval Augmented Generation Assessment) is the most popular open-source framework for evaluating RAG systems. It provides a suite of metrics that work without requiring ground truth labels.

The Core Metrics

Faithfulness — Does the answer contain hallucinations?

The metric works like this: an LLM extracts all claims made in the answer. Then it checks each claim against the retrieved context. If a claim isn't supported by the context, it's hallucinated.

Score: 0–1, where 1 means "everything is supported by the retrieved context."

Why it matters: This catches the sneaky case where your LLM generates grammatically perfect nonsense.

Answer Relevancy — Is the answer actually relevant to the user's question?

Instead of asking a human "is this relevant?", RAGAS generates multiple synthetic questions from the answer using the LLM, then measures how similar those questions are to the original query.

Score: 0–1, where 1 means "the answer is directly answering what was asked."

Why it matters: An answer can be faithful to the context and completely miss what the user wanted.

Context Precision — Are the most useful chunks ranked first?

When you retrieve 10 documents, is the most relevant one at position 1? Or buried at position 7? This metric measures whether the retriever ranked things in the right order.

Score: 0–1, where 1 means "every retrieved chunk is relevant."

Why it matters: If your LLM has to wade through junk to find useful context, it'll get confused or hallucinate.

Context Recall — Did you retrieve everything needed?

This asks: given the correct answer, how much of the supporting context did the retriever actually find?

Score: 0–1, where 1 means "you got all the context needed to answer correctly."

Why it matters: If key context is missing, your LLM can't answer well, no matter how good it is.

Putting It Together

You're not checking one metric. You're checking four dimensions of a single evaluation:

Faithfulness measures hallucinations → low faithfulness = your LLM is making things up
Answer relevancy measures understanding the question → low relevancy = wrong answer, right format
Context precision measures retriever ranking → low precision = retriever is mixing junk with gold
Context recall measures completeness → low recall = retriever missed important context

A healthy RAG system has all four scores high. If faithfulness is low, you have a hallucination problem. If recall is low, your retriever is weak.

Beyond RAGAS: LLM-as-Judge

RAGAS is great for RAG systems specifically. But what if your system is more general? What if you're not using retrieval?

That's where LLM-as-Judge comes in.

The idea is simple: use a powerful LLM (like GPT-4) to evaluate the outputs of another LLM, prompted to assign scores on dimensions like helpfulness, correctness, faithfulness, or safety.

Judge prompt (simplified):
"You are an expert evaluator. The user asked: [QUERY]
The system responded: [ANSWER]
Rate the response from 1-10 on correctness, helpfulness, and truthfulness.
Explain your reasoning."

Pros:

No ground truth needed
Works for any task (not just RAG)
Can evaluate complex, nuanced quality
Aligns with human judgment (85%+ agreement with human raters on GPT-4 judgments)

Cons:

Costs money (you're calling an LLM to evaluate another LLM)
Can inherit biases from the judge (GPT-4 has position bias, verbosity bias, self-enhancement bias)
Prompt wording matters a lot—small changes in phrasing can shift scores by 10-15%

Pro tip: Use Chain-of-Thought prompting with your judge. Ask it to explain its reasoning step-by-step before assigning a score. This improves reliability by 10-15% and gives you a debuggable reasoning trail.

Hallucination Detection: The Hard Problem

Here's the truth: hallucinations are hard to detect automatically.

Your LLM generates a paragraph that sounds completely plausible. It cites no sources (because it made it up). It's grammatically perfect. How do you know it's wrong without checking every fact manually?

Recent approaches:

Self-Consistency Methods (SelfCheckGPT):

Generate the same answer multiple times with different random seeds
If the answer is consistent across generations, it's probably faithful
If it varies wildly each time, it's probably hallucinated
This works because factual claims are stable; hallucinations drift

Token Probability Methods:

Look at the model's internal confidence scores during generation
If the model assigns low probability to its own words, something's off
This doesn't always work—some hallucinations are high-confidence

Supervised Detection:

Train a detector on labeled hallucination data
Feed it hidden state representations from the LLM
Let it predict whether a claim is hallucinated
Works well in-domain; requires new training for new domains

The Honest Answer: There's no silver bullet. You need multiple approaches:

Faithfulness metric to catch unsupported claims
Self-consistency checks for flagrant hallucinations
Human spot-checking on high-stakes domains
Reference-based metrics (comparing output to ground truth) when you have labels

Common Evaluation Mistakes

Mistake 1: Relying on a Single Metric

"Our faithfulness score is 0.92—we're good!"

No. Faithfulness only tells you about hallucinations. Your answer could be:

Faithful but irrelevant (addresses the wrong question)
Faithful and relevant but missing half the context

Evaluate all dimensions. If any dimension is weak, you have a problem.

Mistake 2: Gaming the Metrics

You optimize for high RAGAS scores, so you make your retrieved context smaller (fewer chunks = easier for the LLM to be faithful). Now your scores are great, but your answers miss important details.

Or you use a judge that's biased toward verbose, confident-sounding answers, so your system generates fluff.

The trap: High metrics ≠ good product. You still need human evaluation on real user queries.

Mistake 3: Forgetting About Domain Shift

You evaluate your system on one domain (e.g., Python tutorials) and get great scores. You ship it to production for a different domain (e.g., medical advice). Suddenly users report hallucinations.

This happens because:

Your training data was skewed toward one domain
Your evaluation framework was calibrated on one domain
The LLM's behavior changes in new domains

Always evaluate on representative samples from your actual use case.

Mistake 4: Ignoring the Prompt

LLM judges are incredibly sensitive to how you phrase the evaluation prompt.

"Is this answer correct?" gets different results than "Is this answer helpful and accurate?"
"Rate 1-10" gets different results than "Rate excellent/good/fair/poor"

Test different prompt wordings and see which ones align with your actual needs.

Putting It Into Practice

Here's a lightweight evaluation workflow for your RAG system:

Collect 50-100 real test queries from your users or domain experts
Generate answers using your system
Run RAGAS metrics on all of them
- Calculate mean faithfulness, relevancy, precision, recall
- Flag any queries with low scores
Spot-check the flagged queries manually
- Read the answer and context
- Verify if the metric agrees with your judgment
Iterate — improve your retriever, prompt, or model based on what you find

Tools to use:

RAGAS (open source, free, works with any LLM via API)
DeepEval (Python library, supports RAGAS + custom metrics)
Langfuse (LLM observability platform with built-in LLM-as-judge)
Confident AI (commercial, but focuses on evaluation workflows)

The Real Win: Debugging

Here's the secret nobody tells you: the real value of metrics isn't the score. It's the debugging information.

When your faithfulness score is 0.65, you know where to look: the answer contains unsupported claims. Start examining those claims.

When your context recall is 0.4, you know your retriever is missing stuff. Debug the retriever, not the LLM.

When answer relevancy is low but everything else is high, you know your prompt is asking the wrong question.

Metrics are a map. They point you toward the problem. But you still have to solve it.

Next Steps

Pick one metric that matters to your system (probably faithfulness for RAG)
Set a threshold (e.g., "we want 0.8+ on all metrics")
Evaluate your current system
When scores are low, debug instead of tweaking prompts blindly
Repeat

Start small. Don't build a 500-metric evaluation dashboard on day one. Evaluate the dimensions that matter most to your users, and add more metrics as you grow.

And yes, you still need humans. Metrics catch patterns and point you toward problems. But someone has to verify that the metrics are actually measuring what you care about.

Because in the end, "looks good to me" scales to maybe 100 queries before it breaks. Metrics scale to 100,000. And human judgment backed by metrics? That actually works in production.

Your turn: what does your LLM system get wrong most often? Is it hallucinations, missing context, or something else? Metrics can help you find out.

Author: Shibin

How DeepSeek R1 Shocked the World (And Why It Matters to You)

Thousand Miles AI — Fri, 06 Mar 2026 05:41:43 +0000

The underdog story that disrupted AI. 671B parameters, $6M budget, MIT license. How a Chinese startup beat the giants.

January 20, 2025. A Tuesday morning in Hangzhou, China.

A small AI lab called DeepSeek dropped something that made the entire AI industry go quiet.

They released DeepSeek-R1: a 671-billion-parameter reasoning model that matched or exceeded OpenAI's o1 on most tasks. On AIME (a difficult math competition for 10th-12th graders), R1 scored 79.8%. On MATH (a dataset of problems from mathematics competitions), it scored 97.4%. These are the kinds of numbers that previously belonged only to expensive, closed models.

Here's the kicker: they built it in two months for less than $6 million.

And they open-sourced it under the MIT license.

For free.

For commercial use.

The entire AI industry had assumed that building frontier models required billions of dollars, massive research teams, and years of development. DeepSeek just proved them wrong.

This is the story of how the underdog broke the game. And why, if you're a student or early-career developer in India in 2026, you should care deeply.

Why Should You Care?

Let's be real: if you're learning AI or building with language models right now, you probably assume that the best tools cost money and come from Silicon Valley.

DeepSeek-R1 changes that entire equation.

Suddenly, the best reasoning model in the world is free. The weights are public. The code is public. The architecture is documented. You can run it locally. You can fine-tune it. You can build on top of it.

This isn't a small thing. This is a paradigm shift.

In 2025, if you wanted to use frontier AI models, your options were:

Pay OpenAI $20/million tokens for GPT-4o
Pay Anthropic for Claude
Use open models that were 1-2 years behind the frontier

By early 2026, you can:

Use DeepSeek-R1 via API for 95% cheaper than GPT-4o
Download the weights and run it on your own hardware
Fine-tune it on your own data
Contribute improvements back to the community

For students and early-career developers? This is liberation.

The Narrative: How Did a Chinese Startup Do This?

To understand why DeepSeek's achievement is shocking, you need to understand the before-times.

The AI scaling narrative went like this:

2018-2022: "Models need more data and more compute. Those with the most resources win."

This was true. Google, OpenAI, Meta poured billions into training. The scaling laws held. Bigger compute = better models. The end.

2023-2024: The assumption became dogma. "You need a billion-dollar budget to compete."

OpenAI's Sora. Google's Gemini. Meta's Llama. These required massive computational resources. The era of small labs was over. Only Big Tech could innovate.

Then DeepSeek whispered: "What if we were... actually optimizing things?"

They looked at the scaling curves and noticed something. The industry was optimizing for training speed, not training cost. We were throwing compute at problems because we could afford to. What if we optimized for efficiency instead?

Enter: Mixture of Experts (MoE) architecture.

Instead of activating all 671 billion parameters for every token, DeepSeek's model only activates 37 billion parameters. The rest sit dormant until needed. Different "experts" handle different types of problems.

Think of it like having a massive library where you don't need to read every book for every question. You route questions to the expert who knows the most about that topic.

The result? All the capability of a massive model with 5% of the compute cost.

The Training Secret: Reinforcement Learning First

But there's another part to the DeepSeek story that's almost as important.

Most models train in two stages:

Supervised Fine-Tuning (SFT): Train on high-quality examples. "Here's how a smart human would answer this question."
Reinforcement Learning (RL): Improve through reward signals. "Did the model do better?"

The industry sequence: SFT first (to establish baselines), then RL (to polish).

DeepSeek said: "What if we do massive RL without SFT?"

They called it DeepSeek-R1-Zero. Train the model primarily through reinforcement learning, without supervised examples telling it the "right" way to reason.

The model had to figure out reasoning from first principles.

And it worked. Remarkably well.

Here's why this matters: It suggests that reasoning isn't taught—it's incentivized. Give a model the right reward signal and a search space to explore, and it will discover how to reason. You don't need humans showing it examples of perfect reasoning.

This is a fundamental insight. It changes how we think about training future models.

And a small lab in Hangzhou figured it out before the multi-billion-dollar labs in San Francisco and Mountain View.

The Shock to the System

When DeepSeek released R1, the AI industry had several very uncomfortable realizations:

Realization 1: The scaling compute wasn't about fundamentals, it was about shortcuts.

Google, OpenAI, Meta could afford to throw compute at problems. They did. It worked because throwing compute at problems does work. But it's not the only way. DeepSeek showed that intelligence-per-dollar isn't a function of billions spent. It's a function of creativity.

Realization 2: Open source was still winning, just slowly.

The largest open-source model (Meta's Llama 3) was pretty far behind GPT-4o. By releasing DeepSeek-R1 under MIT, the open-source community instantly jumped ahead of Llama. Now researchers everywhere could work with frontier-quality models.

Realization 3: Geography doesn't matter anymore.

In 2020, the narrative was: "Silicon Valley and Big Tech have the network, the talent, the funding. You can't compete from outside." DeepSeek disproved this. A lab in China, building autonomously, shipped something that matched or exceeded everything from Silicon Valley.

Realization 4: Cheap inference changes everything.

When your model costs 95% less to run, you can afford to run it in places that previously couldn't use frontier AI. You can fine-tune it cheaply. You can experiment. The barrier to entry drops from millions to thousands.

What DeepSeek R1 Actually Does

Let's talk about the actual capability. What makes R1 special?

Unlike most language models, R1 is a "reasoning model." When you ask it a hard question, it doesn't just stream an answer. It thinks out loud.

User: Solve this differential equation: dy/dx = 2x + 1

DeepSeek R1: [THINKING]
I need to solve dy/dx = 2x + 1
This is a simple differential equation.
I can integrate both sides...

[RESPONSE]
To solve dy/dx = 2x + 1:
Integrate both sides with respect to x:
y = ∫(2x + 1)dx = x² + x + C

It shows its work. It reasons through problems step-by-step. For mathematical problems, coding problems, and complex logic, this is dramatically more accurate than models that try to stream answers directly.

On benchmarks:

AIME (competition math): 79.8% (o1 was 92%, R1 is closer than expected)
MATH (math competition problems): 97.4% (frontier performance)
MMLU (broad knowledge): 90.8% (GPT-4o performance)
Coding: Strong performance on competitive programming

But here's the real revelation: R1 is open-source. Public weights. MIT license. Run it anywhere.

Meanwhile, OpenAI's o1? Closed. $20/million tokens via API. You can't see how it works. You can't run it yourself.

For students and researchers, this is revolutionary.

The Ripple Effects in 2026

By February 2026 (just a month later), the ecosystem had already transformed:

Wave 1: Distillation Models

Teams immediately started distilling R1 into smaller models. "If we can capture 80% of R1's reasoning ability in a 7B or 13B model, we can run it on laptops."

By month two, we had open-source reasoning models at various sizes. Not as good as R1, but genuinely useful. All free.

Wave 2: Rapid Iteration

The entire open-source community moved faster. Researchers published papers on improving reasoning. Teams fine-tuned R1 for specific domains (medical reasoning, code generation, creative writing). Without waiting for OpenAI to release the next model, people were already building on top of R1.

Wave 3: Cost Collapse

DeepSeek's API (95% cheaper than OpenAI) forced the entire industry to reconsider pricing. By January 2026, everyone was dropping prices. What cost $20 per million tokens now costs $0.50.

For startups? For students? For countries where AI was previously inaccessible? Suddenly affordable.

Wave 4: Chinese AI Ecosystem

The narrative shifted. China wasn't just copying Western AI. They were innovating faster, cheaper, better. Investment flowed into Chinese AI labs. The ecosystem that produced DeepSeek started producing other frontier models.

Why This Matters If You're Starting Your Career

Here's the real-world impact for you:

1. You Can Afford Frontier AI

If you're building a startup or learning AI, you can now use the best reasoning models for a fraction of what it cost six months ago. Your laptop-based experiments aren't competing on a budget—they're competitive with well-funded labs.

2. You Can Run Models Locally

You don't need cloud access. You don't need to depend on API availability. Download DeepSeek-R1 and run it on your hardware. This is freedom. This is sovereignty over your tools.

3. Open Source Is Winning Again

For the last few years, the frontier was in closed models. Suddenly, open-source is at the frontier. This means:

You can read the code
You can contribute improvements
You can build on top of it without worrying about API deprecations
You can fine-tune it for your use case

4. Geography Doesn't Matter

DeepSeek proved you don't need to be in Silicon Valley to build world-class AI. You can be in Bangalore. You can be in Manila. You can be in Lagos. With internet and talent, you can compete at the frontier.

5. Thinking About Fundamentals Pays Off

DeepSeek's success wasn't about throwing money at the problem. It was about creative thinking. Mixture of Experts. Different training approaches. Understanding the math deeply.

For you starting your career: the advantage isn't money. It's understanding. Deep knowledge of fundamentals beats budget every time. DeepSeek proved it.

The Plot Twist: What Happens Next?

This is February 2026. DeepSeek R1 was released one month ago.

The question now is: does this become the standard? Does the AI industry permanently shift toward efficiency and open-source? Or was this a one-time breakthrough?

My bet? DeepSeek made something irreversible happen.

Once you've proven that you can build frontier models for 1/100th the cost, you can't un-prove it. Every lab will now try to optimize for efficiency. Every competitor will need to match those costs.

The golden age of billion-dollar training runs might be over.

But here's the thing that worries the incumbents: DeepSeek also proved that raw spending doesn't guarantee dominance. Strategy, creativity, and understanding fundamentals matter more.

For the open-source community? For students? For people outside Silicon Valley?

This changes everything.

You're not competing against a wall of money anymore. You're competing against intelligence, creativity, and determination.

And those are things you can build.

Sign-Off: The Underdog Story Isn't Over

DeepSeek's story is an underdog narrative in real-time. A small lab. A focused mission. A creative solution. And they won.

Not by playing the game the incumbents set up. But by changing the rules entirely.

If you're starting your career in 2026, take note. The lesson isn't "use DeepSeek." The lesson is: "the people who win aren't the ones with the most resources. They're the ones who think differently."

DeepSeek thought about mixture of experts when everyone else thought about scale. They trained with RL-first when everyone else did SFT-first. They open-sourced when everyone else locked things down.

Different thinking wins.

Use DeepSeek-R1 if it serves your project. Build on top of open-source reasoning models. Fine-tune them for your domain. Contribute improvements back to the community.

But more importantly: think about where the incumbents are wrong. Where are they assuming things that aren't true? Where could you approach the problem differently?

That's where the next decade of innovation lives.

And it might come from someone sitting in their apartment in Bangalore, not from some well-funded lab in San Francisco.

The age of gatekeeping AI is over.

Now it's just about being creative.

References & Further Reading

Author: Shibin

Context Engineering Is the New Prompt Engineering

Thousand Miles AI — Fri, 06 Mar 2026 05:36:36 +0000

How CLAUDE.md files and structured context are transforming AI coding. One file to rule them all.

Last year, prompt engineering was the hot skill. Craft the perfect prompt. Use the right magic words. Add "step by step." Get better results.

That era is over.

In 2026, the teams getting dramatically better results from AI aren't tinkering with prompts. They're architecting context. They're creating .claude/CLAUDE.md files that sit at the root of their projects. They're encoding their entire system architecture, coding standards, and project rules into a single structured document that the AI reads before every response.

It's not flashier than prompt engineering. It's just more effective.

Why Should You Care?

Picture this: You're working with Claude on a feature. You give it a task. It generates code that's syntactically perfect but doesn't match your project's patterns. It uses styled-components when your codebase uses Tailwind. It puts business logic in components when you have a strict repository pattern. It ignores your testing standards.

So you say: "We use Tailwind, not styled-components."

Claude fixes it.

You say: "Put business logic in repositories, not components."

Claude adjusts.

You say: "Add unit tests using Vitest."

Claude adds them.

You've just engaged in five rounds of back-and-forth context negotiation. You've burned tokens. You've wasted time. And you're frustrated.

Now imagine a different scenario. The project has a CLAUDE.md file. It contains:

Your exact tech stack
Your styling approach and typography system
Your folder structure and where to put things
Your testing framework and where tests live
Your database access patterns
Your component rules
Your exact conventions for naming, formatting, and architecture

Claude reads this file once at the start. Now when you ask for a feature, it generates code that fits your project perfectly. First try.

Recent studies show that structured context engineering reduces hallucinations by 60% and improves task completion accuracy by 45% compared to prompt engineering alone. This isn't magic wording. This is information architecture.

The Context Engineering Manifesto

Context engineering is broader than prompt engineering. It's not just about what you say—it's about what the AI knows before you say anything.

Prompt Engineering: "Write a function that validates email addresses"

Context Engineering: The AI reads your CLAUDE.md, which says:

You use TypeScript
All validators go in lib/validators/
Use Zod for validation
Tests go in tests/unit/
Follow existing validator patterns in the codebase
Export both validate* function and type *Error

Now when you ask: "Write an email validator," the AI generates code that already fits perfectly into your system.

Think of it as the difference between giving someone directions ("turn left at the big oak tree") versus giving them a GPS with your entire city mapped out.

Anatomy of a Good CLAUDE.md File

Let me break down what goes into an effective context file. Here's the structure:

1. Project Overview

# Project Name

One sentence what this is. Tech stack. Key constraints.

> Next.js + Tailwind CSS + Supabase + Drizzle ORM

Why? The AI needs to know the boundaries of the project before it starts working.

2. Mandatory Completion Checklist

## MANDATORY: Completion Checklist

DO NOT declare any task complete until:
1. Write tests — Unit/Integration/E2E as applicable
2. Run `npm test` — All tests pass
3. Run `npm run build` — Build succeeds
4. Verify — If any fail, fix it before summarizing

Why? This forces the AI (and you) to have quality gates. Tasks don't end until they're actually done.

3. Rules at a Glance

Quick reference table. Styling approach. Where components go. How to structure files. What frameworks to use for what.

| Situation | Action |
| Page title | `<h1 className="text-heading-32">` |
| Card styling | Direct Tailwind, no component |
| Database queries | Drizzle repositories in `src/lib/repositories/drizzle/` |

Why? The AI can scan this table before generating code and match every pattern.

4. Styling System

Exactly how typography works in your project. What classes exist. When to use what.

## Purpose-Based Typography

- `text-heading-32`: Page titles, 2rem, 600 weight
- `text-heading-24`: Section headers, 1.5rem, 600 weight
- `text-copy-16`: Body text, 1rem, 400 weight, relaxed leading
- `text-label-14`: UI labels, 0.875rem, 500 weight

Why? Otherwise AI generates random class names or uses hardcoded pixel values instead of your system.

5. Component Rules

When to create components. What goes in components. What doesn't.

## DO NOT Create Components For
- Styling only — use Tailwind directly
- Single-use layouts — inline the classes
- Wrapper components

## DO Create Components When
- Behavior/State exists (toggle, form, async actions)
- Used 3+ times identically
- Requires accessibility logic

Why? Without this, you get components for everything. Bloated folders. Prop chaos.

6. Database Access Patterns

Exactly how to access data. Where repositories live. How to structure queries.

## Database — Use Drizzle Repositories

ALL database queries go in `src/lib/repositories/drizzle/`.
Never use Supabase client directly for tables.
ONLY use Supabase client for `supabase.auth.*` and `supabase.storage.*`.

Why? Database patterns are architectural decisions, not suggestions. The AI needs to know the non-negotiables.

7. Testing Structure

Where tests go. What frameworks. What counts as tested.

## Testing — Mandatory

| Change Type | Unit | Integration | E2E |
| New utility | REQUIRED | — | — |
| New API endpoint | REQUIRED | REQUIRED | If user-facing |
| New feature | REQUIRED | If API | REQUIRED |

Why? Different changes need different testing levels. Without this, the AI either over-tests or under-tests.

8. Project Structure

Where things go. Folder conventions. What each folder contains.

## Source Structure

src/
├── components/
│   ├── ui/          # shadcn/ui (behavior-focused)
│   └── constructs/  # Reusable patterns
├── app/
│   └── (protected)/
├── lib/
│   ├── repositories/  # ALL database queries
│   └── utils/

Why? The AI needs to know exactly where to put new code. Without this, files go everywhere.

Context Engineering vs. Prompt Engineering: A Real Example

Let's see the difference in action.

Without Context Engineering

You: "Create a course enrollment component"

Claude generates:

import styled from 'styled-components'

const CourseCard = styled.div`
  padding: 24px;
  border-radius: 8px;
  background: #f5f5f5;
  &:hover { background: #e0e0e0; }
`

export function EnrollButton() {
  return <button>Enroll Now</button>
}

Problems:

Uses styled-components (wrong—you use Tailwind)
No TypeScript types
Component in wrong location
No test file
Hardcoded colors instead of semantic values

You: "We use Tailwind, not styled-components. Put it in components/constructs/. Add types."

Back-and-forth negotiation begins.

With Context Engineering

Your CLAUDE.md says:

- Styling: Use Tailwind directly. No CSS-in-JS.
- Typography: Use purpose-based classes (text-heading-*, text-copy-*)
- Components: Only for behavior. Styling = Tailwind directly.
- Constructs: Reusable patterns go in `components/constructs/`
- Testing: Unit tests in `tests/unit/`, E2E in `tests/e2e/`
- Naming: Kebab-case filenames

You: "Create a course enrollment component"

Claude generates:

// components/constructs/course-enrollment-button.tsx
interface CourseEnrollmentButtonProps {
  courseId: string
  onEnroll: () => Promise<void>
}

export function CourseEnrollmentButton({
  courseId,
  onEnroll,
}: CourseEnrollmentButtonProps) {
  const [pending, setPending] = useState(false)
  // ... implementation using Tailwind
}

Plus a test file. Plus proper types. First try.

The only difference? The AI read your CLAUDE.md.

Building Your Own Context Engineering System

If you're running an AI-heavy project, here's how to start:

Step 1: Audit Your Patterns

Spend a day documenting how your project actually works:

How do you style things?
Where do files go?
What naming conventions do you use?
What's your testing approach?
What frameworks are non-negotiable?

Step 2: Create `.claude/CLAUDE.md`

Write this down. Be specific. Use examples. Include the annoying edge cases that the AI will probably get wrong.

/project-root/
├── .claude/
│   └── CLAUDE.md
├── src/
├── tests/
└── ...

Step 3: Reference It in Your Prompts

Just mention it: "Before you start, check the CLAUDE.md file in .claude/ for project rules and conventions."

Or let the AI infer it. Many modern AI systems automatically scan for context files.

Step 4: Iterate

The first version won't be perfect. When the AI does something wrong, add a rule to CLAUDE.md. When you notice a pattern, document it. Over time, your context file becomes more powerful.

The Teams Winning With Context Engineering

Companies and open-source projects using context engineering in 2026 are:

Shipping faster — First-try accuracy means fewer iterations
Maintaining consistency — The CLAUDE.md becomes a source of truth
Onboarding new team members — Read the CLAUDE.md, understand the project
Training junior developers — Your coding standards are documented, not tribal knowledge

Some teams even use context files for other purposes:

Onboarding documentation — New developers read CLAUDE.md to understand the project
Code review guidelines — The CLAUDE.md is the standard
AI training data — Structure your project around the context file, get better AI results

Sign-Off: Prompt Engineering Was Just the First Stage

Here's what's happened: Prompt engineering was the first wave. "How do I get the AI to understand what I want?"

Context engineering is the second wave. "How do I structure my entire project so the AI automatically understands?"

This isn't about being clever with words. It's about being systematic with information. You're not asking the AI to read your mind. You're handing it a map to your entire codebase's rules.

In 2026, the competitive advantage isn't who can write the best prompts. It's who can structure the clearest context.

Start small. Pick one project. Audit your patterns. Write a CLAUDE.md file. Use it for a week. See how many rounds of back-and-forth disappear.

That's context engineering. And it's the present, not the future.

References & Further Reading

Author: Shibin

Running LLMs on Your Laptop Without a $10K GPU

Thousand Miles AI — Fri, 06 Mar 2026 05:32:34 +0000

Practical guide to running production-ready LLMs locally using Ollama, llama.cpp, and quantization. No GPU cluster required.

The Setup: Your Laptop as an AI Powerhouse

You're sitting in your college hostel. Your friend won't stop talking about how they're building this incredible LLM app, but they're stuck because cloud API costs are bleeding them dry. $0.002 per token adds up fast when you're iterating, testing, and frankly, making mistakes.

Then you mention: "I just spun up a 7B model on my MacBook."

Their face. Worth it.

Here's the reality of 2026: you don't need the internet, you don't need Anthropic's API, and you definitely don't need a $10K GPU cluster. You can run production-quality LLMs on your laptop right now, offline, for free.

This isn't a hobby anymore. It's practical.

Why Should You Care?

Cost. Seriously. If you're building a startup, running 100 inference requests against Claude costs you real money. Running the same requests locally costs you electricity and disk space.

Privacy. Everything stays on your machine. Your prompts aren't going to some company's servers. Your data doesn't train their next model. That matters if you're working with sensitive information—medical data, financial models, client projects.

Speed. Local inference is fast. No network latency. No queuing. No rate limits. You can iterate at the speed of thought.

Offline capability. Working on a plane? In a rural area with spotty internet? Your LLM doesn't care. It works anywhere.

Learning. Want to understand how LLMs actually work? Running them locally forces you to think about quantization, memory management, token limits—the details that cloud APIs hide from you.

So here's what we're going to do: you're going to learn how to run real LLMs on your actual laptop, understand what's happening under the hood, and know exactly when to reach for cloud APIs and when to keep it local.

Part 1: The Technology Stack

llama.cpp: The Secret Sauce

At the heart of everything is llama.cpp, a pure C/C++ implementation of LLM inference created by Georgi Gerganov. No dependencies. Just raw efficiency.

What makes it special: it's optimized for consumer hardware. ARM NEON on iOS. Metal on Apple Silicon. AVX/AVX2/AVX512 on x86. Your CPU isn't just supported—it's screaming.

Think of llama.cpp as the production runtime for LLMs. It's fast, it's memory-efficient, and it's the foundation of everything in this post.

Ollama: The User-Friendly Frontend

Ollama is built on llama.cpp but adds a layer of "just use it." You install Ollama, you run a command, and boom—you've got an LLM server running locally.

ollama run mistral

That's it. Ollama handles downloading the model, quantization, all of it. You get a chat interface, you get a local API endpoint at localhost:11434, and you can build on top of it.

GGUF: The Model Format

GGUF stands for "General GPU Format for Universal Models" (though "GPU" is misleading—it works great on CPU too). It's the standard format for quantized LLMs in 2026.

Why GGUF? It's optimized for loading and inference. It stores metadata efficiently. It supports advanced quantization techniques. And crucially: 45,000+ quantized models exist on Hugging Face Hub right now, ready to download and use.

Part 2: Understanding Quantization

Here's where most people's eyes glaze over. Let's fix that.

What's Quantization Actually Doing?

An LLM like Mistral 7B is stored in FP16 (16-bit floating point) by default. That means every number in the model takes 16 bits of memory. With 7 billion parameters, that's roughly 14GB of VRAM.

Quantization is approximation with a purpose. Instead of storing every number precisely, you store it with less precision. A 4-bit version of the same model takes ~3.5GB. A 3-bit version takes ~2.6GB.

"But won't it be worse?" you ask.

Barely. Here's why: neural networks are robust. Small precision loss doesn't matter much. You lose maybe 5-10% of quality when going from FP16 to Q4_K_M (4-bit). The models are trained on data with noise, so they're used to imperfect inputs.

The Quantization Landscape

Q4_K_M is the sweet spot for most people. It's the Goldilocks quantization—balances quality and size perfectly. Your Mistral 7B becomes ~3.5GB, runs on basically any laptop, and you lose almost nothing in quality.

Need faster inference? Drop to Q3_K_M. Need absolute best quality? Use Q6_K or Q8. But start with Q4_K_M. It's magic.

How Models End Up Quantized

When you download a model from Hugging Face marked as "GGUF Q4_K_M", someone has already quantized it. Usually it's the community. You download, you use immediately. No extra work.

If you want to quantize your own model (because you've fine-tuned it, or you found a cool model in FP16), llama.cpp has a quantize tool:

./llama-quantize model.gguf model-q4.gguf Q4_K_M

Takes a few minutes. You now have a quantized version. Use it.

Part 3: Running Your First Local LLM

Installation

macOS:

brew install ollama
ollama serve

Linux:

curl -fsSL https://ollama.ai/install.sh | sh
ollama serve

Windows:
Download the installer from ollama.ai. It handles CUDA/ROCm automatically if you have an NVIDIA GPU.

Running a Model

Open a new terminal:

ollama run mistral

Ollama downloads the model (~4GB for Q4_K_M Mistral 7B). First run takes a minute. Then you get a prompt:

>>> What's quantization?

Type your question. Get your answer. Offline. Instant. No API key.

Using It Programmatically

Ollama starts an API server at http://localhost:11434. You can hit it like any LLM API:

curl http://localhost:11434/api/generate -d '{
  "model": "mistral",
  "prompt": "explain quantization in one sentence",
  "stream": false
}'

Or from Python:

import requests

response = requests.post("http://localhost:11434/api/generate", json={
    "model": "mistral",
    "prompt": "What is quantization?",
    "stream": False
})

print(response.json()["response"])

That's it. You now have a local LLM API. Build on top of it. Use it in your Next.js app. Whatever.

Part 4: What Models Should You Actually Use?

We've tested models on MacBook Pro M3, RTX 4090, and Raspberry Pi. Here's what works:

For Fast Responses (3-5 sec per 100 tokens):

TinyLlama 1.1B — Surprisingly capable. Fast. Good for classification tasks.
Phi-4-mini 3.8B — Microsoft's breakthrough. GPT-3.5-class reasoning from 3.8B parameters. Insane.
Mistral 7B — Still the king. Balanced. Good at everything. 7 seconds per 100 tokens on M3.

For Quality (10-15 sec per 100 tokens):

Mistral Large 34B — If your laptop can handle it (32GB+ RAM). Genuinely good.
LLaMA 3.1 13B — Rock solid. Open weights. Well-supported.
Qwen 2.5 14B — Chinese-trained. Excellent multilingual support. Great reasoning.

For Code Generation:

Mistral Coder 7B-nemo — Purpose-built for coding. Better than Mistral for programming tasks.
DeepSeek Coder 6.7B — Fast. Surprisingly good at complex code.

You can run any of these right now, today, on your laptop.

Part 5: Common Mistakes

Mistake 1: Not understanding memory consumption

The model file size isn't your memory usage. A 7B model in Q4_K_M is ~3.5GB on disk, but it uses ~8-10GB RAM when running. Your operating system needs RAM too. You need headroom. If your laptop has 16GB total RAM, you can comfortably run a 7B model. If you have 8GB, stick to 3-4B models.

Mistake 2: Not using the right quantization

"I want the best quality" so you download Q8_K. Then it's slow and uses tons of RAM. Try Q4_K_M first. Measure the quality. Only upgrade if you need to.

Mistake 3: Forgetting about context window

LLMs have a maximum context length. Mistral 7B has 32k tokens. You can't feed it a 100,000 token document and expect it to process all of it. Use summarization or retrieval-augmented generation (RAG) to feed the model only relevant context.

Mistake 4: Not batch-processing

Need to run inference on 1,000 prompts? Don't do it one at a time. Batch them. Local inference is fast—batch processing makes it even faster.

Mistake 5: Ignoring GPU options

If you have an NVIDIA GPU, tell Ollama. It'll use CUDA and be 5-10x faster. Ollama auto-detects, but verify it's using your GPU:

ollama list  # Shows GPU allocation

Part 6: When to Stay Local vs. When to Go Cloud

Stay Local:

Development & iteration (you're paying per token on cloud)
Privacy-sensitive work (medical, financial, proprietary)
Offline applications
Running experiments (you control when you pay)
Building local features that don't scale globally

Go Cloud:

Production serving thousands of users (you need enterprise scaling)
Advanced reasoning (Claude Opus / GPT-4 are still better)
When latency doesn't matter (batch processing external APIs)
Prototyping new capabilities (test expensive models cheaply)

The future is hybrid. Run Mistral locally for 95% of your tasks. Send hard problems to Claude. Save money. Get better results.

Next Steps

Download Ollama from ollama.ai. 5 minutes.
Run ollama run mistral in your terminal. Try talking to it.
Read your laptop's specs. How much RAM? Do you have a GPU? This determines what models you can run.
Build something. Query the local API from a script. Create a simple chat interface. Add RAG with local embeddings.
Benchmark. Time a cloud API call vs. local inference. See the cost difference over a month.

If you're building anything AI-powered—and in 2026, what isn't?—running models locally is a superpower you actually have access to right now.

Sign-Off

Remember that friend who was bleeding money on API tokens? You can be the one who whispers: "I just spun up a 7B model on my laptop. What were you saying about API costs?"

It's not some future thing. It's today. Your actual laptop. Right now.

Go run something.

What Are Reasoning Models and Why Do They Think Before Answering?

Thousand Miles AI — Fri, 06 Mar 2026 05:32:29 +0000

o1, o3, DeepSeek R1 — a new breed of LLMs that literally pause to think. But what does 'thinking' mean for a model? Inside thinking tokens, chain-of-thought training, and why this changes everything about how LLMs solve problems.

What Are Reasoning Models and Why Do They Think Before Answering?

Regular LLMs blurt out answers. Reasoning models stop, think, check their work, and then answer. The difference is bigger than you'd expect.

The Model That Argued With Itself

Here's something wild. If you give DeepSeek R1 a tricky math problem and watch its thinking process (which it shows you, unlike most models), you'll see something that looks almost... human. It starts with an approach. Gets halfway through. Realizes something doesn't add up. Literally writes "Wait, that's not right" to itself. Backtracks. Tries a different approach. Checks the answer. Then gives you the final result.

It's not performing for you. These are internal reasoning tokens — the model's scratch pad. Some models hide this thinking process. R1 shows it to you in full. And it's genuinely fascinating to watch a model second-guess itself, catch its own errors, and course-correct.

This is what makes reasoning models different from everything that came before. Standard LLMs generate answers one token at a time, left to right, committing to each word as they go. They don't plan ahead. They don't check their work. Reasoning models add a phase before the answer where they think through the problem step by step — and that simple addition dramatically improves performance on math, coding, logic, and scientific reasoning.

Why Should You Care?

Two reasons. First, reasoning models are quickly becoming the go-to choice for any task that requires multi-step logic — coding, data analysis, math, complex question answering. If you're building AI-powered tools, knowing when to use a reasoning model versus a standard one is a practical skill.

Second, the techniques behind reasoning models — chain-of-thought training, reinforcement learning without human supervision, knowledge distillation — represent a genuine shift in how AI research works. Understanding these concepts puts you ahead of the curve, whether for interviews, research, or building your own systems.

Let Me Back Up — What's Actually Different?

Regular LLMs (GPT-4, Claude Sonnet, Gemini) take your input and generate output directly. Ask a math question, and the model starts writing the answer immediately. It's fast, but it's also impulsive — the model commits to its first approach without considering alternatives.

Reasoning models add an intermediate step: a thinking phase where the model generates chain-of-thought tokens before producing the final answer. Think of it as the difference between a student who immediately scribbles an answer on an exam versus one who works through the problem on scratch paper first.

Standard LLMs go straight to the answer. Reasoning models generate thinking tokens first, then verify, then answer.

The key insight that made this work: more thinking tokens = better answers. Researchers found that performance improves predictably with the number of tokens the model spends "thinking." This is called test-time compute scaling — spending more computation during inference (when the model generates a response) rather than only during training.

Okay, But What Does "Thinking" Actually Look Like?

Let's make this concrete. When DeepSeek R1 encounters a question, it wraps its reasoning in <think> tags. Here's a simplified version of what that looks like internally:

User: What's the sum of the first 100 prime numbers?

<think>
I need to find the first 100 prime numbers and add them up.
Let me start listing them: 2, 3, 5, 7, 11, 13...
Wait, instead of listing all 100, let me recall that the 100th prime is 541.
The sum of the first 100 primes is a known value: 24,133.
Let me verify - the first few: 2+3+5+7+11 = 28. That checks out with known partial sums.
I'm confident the answer is 24,133.
</think>

The sum of the first 100 prime numbers is 24,133.

Notice what's happening: the model explores an approach (listing primes), reconsiders (too slow), pivots to a more efficient strategy, verifies its answer against known data, and then produces the final output. This self-correction during generation is what makes reasoning models special.

With OpenAI's o1 and o3, you see something similar — but they use a "private chain of thought." The model still thinks through the problem, but you only see a summarized version. DeepSeek R1's full transparency is one of its distinguishing features.

How They're Built — The Training Behind Reasoning

Here's where it gets technically interesting. There are two main approaches to building reasoning models, and they reveal very different philosophies.

The OpenAI Approach: Reinforcement Learning on Curated Data

OpenAI hasn't published full details on o1/o3's training, but the broad strokes are known. They use reinforcement learning (RL) to train the model to produce better chain-of-thought reasoning. The model generates reasoning traces, those traces are evaluated (did they lead to correct answers?), and the model is rewarded for reasoning patterns that work.

The reasoning process is private — OpenAI chose not to expose the raw thinking tokens. You see a summary of the reasoning, not the full internal monologue. This is a deliberate design choice, likely for both user experience and competitive reasons.

The DeepSeek Approach: RL from Scratch

DeepSeek took a bolder path. They published their full methodology, and it's remarkable.

Phase 1 — R1-Zero (pure RL, no human examples): They took a base model and applied reinforcement learning directly, without any human-written chain-of-thought examples. They just rewarded the model for getting correct answers and penalized it for wrong ones. The model discovered chain-of-thought reasoning on its own.

This is the mind-blowing part: nobody taught R1-Zero to "think step by step." It learned that writing out intermediate reasoning led to better rewards. It independently developed self-verification — checking its own work. It even had what the researchers called an "aha moment," where it suddenly started using the word "Wait" during its reasoning, marking a distinct shift to more self-reflective thinking.

Phase 2 — Polishing: R1-Zero worked but had issues — repetitive reasoning, language mixing, poor readability. So they added supervised fine-tuning with curated examples, followed by another round of RL for human preference alignment. This produced the final DeepSeek R1.

DeepSeek R1's training pipeline: pure RL discovers reasoning, supervised training polishes it, distillation spreads it to smaller models.

The Distillation Trick

One of DeepSeek's most impactful contributions: they showed you can take the reasoning patterns learned by a massive 671B parameter model and distill them into much smaller models (1.5B to 70B parameters). These distilled models perform remarkably well — a 14B distilled model can outperform many full-sized models on reasoning benchmarks.

This means you don't need a massive model to get reasoning capabilities. The thinking patterns are transferable. That's huge for students and developers working with limited resources.

The Architecture Difference: Dense vs. Sparse

There's an interesting architectural split between the major reasoning models.

OpenAI's o3 uses a dense transformer — all parameters are active for every token. This is computationally expensive but straightforward.

DeepSeek R1 uses a Mixture-of-Experts (MoE) architecture. Of its 671 billion total parameters, only about 37 billion activate for any given token. The rest sit idle. It's like having a team of 20 specialists, but only sending 2–3 of them to handle each task. This makes R1 dramatically cheaper to run despite having more total parameters.

Mistakes That Bite — Common Misunderstandings

"Reasoning models are always better." Not true. For simple tasks — quick Q&A, summarization, casual conversation — standard models are faster, cheaper, and equally accurate. Reasoning models shine on complex, multi-step problems. Using o3 to answer "What's the capital of France?" is like hiring a math PhD to calculate a restaurant tip.

"More thinking tokens always helps." There's a point of diminishing returns. Some problems don't benefit from more thinking — the model just generates redundant reasoning that wastes tokens and money. o3-mini offers three reasoning levels (low, medium, high) for exactly this reason: match the thinking effort to the problem difficulty.

"The thinking tokens are just the model talking to itself." It's more structured than that. The thinking phase includes specific learned behaviors: problem decomposition, hypothesis generation, self-verification, and error correction. These aren't random ruminations — they're patterns the model learned lead to correct answers.

Now Go Break Something

Want to experience the difference firsthand?

Try DeepSeek R1 through their API or web interface — it shows the full thinking process. Give it a tricky logic puzzle and watch it reason through it.
Compare with a standard model on the same problem. Ask GPT-4 or Claude a multi-step math problem, then ask R1. Compare the reasoning quality and accuracy.
Explore distilled versions. The DeepSeek-R1-Distill-Qwen-14B and DeepSeek-R1-Distill-Llama-8B models are on Hugging Face. You can run these locally and get reasoning capabilities on your own machine.
Read the DeepSeek R1 paper — search for "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning." It's published on arXiv and is one of the most accessible AI research papers of 2025.
Search for "The Illustrated DeepSeek-R1" by Jay Alammar — he does visual breakdowns of AI architectures that are incredibly beginner-friendly.

Remember that model arguing with itself — writing "Wait, that's not right" and backtracking mid-thought? That's not a gimmick. It's the result of a model that learned, through pure reinforcement, that slowing down and checking its work leads to better answers. Reasoning models don't know more than standard LLMs. They just take a breath before answering. And that breath — those thinking tokens — turns out to be one of the most powerful improvements in language model history.

Author: Shibin

The 'Lost in the Middle' Problem — Why LLMs Ignore the Middle of Your Context Window

Thousand Miles AI — Fri, 06 Mar 2026 05:32:02 +0000

You stuffed all the right documents into the prompt. The LLM still got the answer wrong. Turns out, language models have a blind spot — and it's right in the middle. Here's the research behind it and what you can do.

The "Lost in the Middle" Problem — Why LLMs Ignore the Middle of Your Context Window

Your LLM has a 128K context window. It can read a novel in one go. But it still misses the one paragraph that matters — because it was in the middle.

The Perfect Retrieval That Still Failed

Here's a scenario that frustrates every RAG developer at some point. You've built a solid pipeline. Your retriever returns five relevant chunks, ranked by relevance. The correct answer is sitting right there — chunk #3, smack in the middle of the context. You've done everything right.

The LLM reads all five chunks, generates a confident response, and... gets it wrong. It pulled information from chunk #1 and chunk #5, blended them together, and produced something that sounds plausible but misses the actual answer. The evidence was right in front of it. It just didn't look at it carefully enough.

You're not imagining this. It has a name: the "lost in the middle" problem. And it's backed by one of the most cited papers in LLM research from 2023, with follow-up work from MIT in 2025 that finally explained why it happens at an architectural level.

Why Should You Care?

If you're building anything that puts multiple pieces of information into an LLM's context — RAG systems, multi-document summarization, long-form analysis — this bias directly affects your output quality. And the bigger your context window, the worse it can get.

This is also the kind of research-backed knowledge that separates strong candidates in AI interviews. Anyone can explain what attention is. Explaining why attention is systematically biased by position and what to do about it — that's a different level.

Let Me Back Up — What the Research Found

In 2023, researchers from Stanford, UC Berkeley, and Samaya AI published a paper titled "Lost in the Middle" that tested how well LLMs use information at different positions in their context window. They ran a simple experiment: give the model a set of documents where only one contains the answer, and vary where that document appears — beginning, middle, or end.

The results showed a clear U-shaped performance curve. When the relevant document was at the very beginning of the context, accuracy was high. When it was at the very end, accuracy was also high. But when it was in the middle? Accuracy dropped — sometimes dramatically.

This wasn't a quirk of one model. They tested multiple LLMs across different architectures and sizes, and the pattern held consistently. Language models pay the most attention to the beginning and end of their context, and systematically under-attend to the middle.

The U-shaped attention curve: LLMs attend strongly to the beginning and end of context, with a blind spot in the middle.

Okay, But Why? — The Architecture Behind the Bias

For two years after the original paper, the "why" was unclear. People noticed the pattern but couldn't pinpoint the cause. Was it training data? Model size? Prompt format?

In 2025, MIT researchers cracked it open. They identified two architectural causes:

Cause 1: Causal Attention Masking

Transformer models use something called causal masking in their attention mechanism. This means each token can only attend to tokens that came before it — not after. It's how the model generates text left-to-right.

Here's the subtle problem: tokens at the beginning of the context get attended to by every subsequent token. Token #1 is visible to token #2, #3, #4... all the way to the end. Token #500, sitting in the middle, is only visible to tokens #501 onward. This means earlier tokens accumulate more "attention weight" across the model, simply because they have more opportunities to be attended to.

It's not that the model decides the beginning is more important. The architecture makes it structurally easier to attend to earlier tokens. The bias is baked into the attention mask itself.

Cause 2: Positional Encoding Decay

Modern LLMs use positional encodings — typically Rotary Position Embedding (RoPE) — to give the model a sense of token order. RoPE introduces a distance-based decay: tokens that are far apart have their attention scores naturally reduced.

For tokens at the end of the context (where the model generates its response), nearby tokens (also at the end) have strong attention signals, and very early tokens also maintain attention through a mechanism called "attention sinks." But middle tokens? They're too far from the beginning to benefit from the primacy effect and too far from the end to benefit from recency. They're in a dead zone.

The Human Parallel

Here's what makes this even more interesting: this mirrors a well-known phenomenon in human psychology called the serial position effect. When people are asked to remember a list of items, they recall the first items (primacy effect) and the last items (recency effect) much better than items in the middle.

LLMs weren't designed to mimic human memory. But through the architecture of attention mechanisms and training on human-generated text, they've developed a strikingly similar bias. Whether this is a bug or a feature of learning from human data is still debated.

Three contributing factors: structural attention bias, positional encoding decay, and training data patterns.

What Can You Actually Do About It?

Knowing the problem is half the battle. Here are practical mitigations that work in production systems:

1. Strategic Document Ordering

The simplest fix: don't put your most important information in the middle. In RAG systems, place your highest-confidence retrieved documents at the beginning and end of the context. Put lower-ranked documents in the middle. You're not fighting the bias — you're working with it.

Specifically: if you retrieve 5 chunks ranked by relevance, arrange them as [rank 1, rank 4, rank 5, rank 3, rank 2] — best at the start, second-best at the end, least important in the middle.

2. Reduce the Number of Retrieved Documents

More context doesn't always mean better answers. If you're retrieving 20 chunks when 5 would suffice, you're creating more middle ground for information to get lost in. Be surgical: use a reranker to select the top 3–5 most relevant chunks and discard the rest. Less noise means less middle to ignore.

3. Prompt Compression

Instead of dumping raw chunks into the context, compress them first. Extract only the sentences or facts that are relevant to the query and assemble a tighter, shorter context. When there's less total content, there's less of a middle for information to hide in.

4. Explicit Instruction

Sometimes the blunt approach works: tell the model to pay attention to all parts of the context. Prompts like "Carefully consider ALL of the provided documents, especially documents that appear in the middle" can measurably reduce the bias. It doesn't eliminate it, but it helps.

5. Multi-Pass Extraction

For critical applications, run multiple passes. First pass: ask the model to extract relevant facts from each document independently. Second pass: ask it to synthesize those facts into an answer. By processing documents individually first, you avoid the position bias entirely — each document gets the model's full attention.

Mistakes That Bite

"Bigger context windows solve this." They don't. The 2023 paper showed the U-curve exists even in models with context windows of 4K, 16K, and 32K tokens. Research from 2025 confirmed it persists in models with 128K+ windows. Bigger windows mean more middle, which means more room for information to get lost.

"This only matters for RAG." It affects any task that puts multiple pieces of information into the context — summarization, question answering over multiple documents, multi-turn conversations where important information was mentioned 20 messages ago. If you're using more than a few hundred tokens of context, this bias applies.

"Newer models have fixed this." Some improvements have been made. Techniques like Multi-scale Positional Encoding (Ms-PoE) and attention calibration can reduce the bias without retraining. But as of 2026, no production model has fully eliminated position bias. It's structural to how transformers work.

Now Go Break Something

Want to see this bias for yourself? Here's a simple experiment:

Create a list of 10 facts. Embed the answer to a specific question as fact #5 (the middle).
Ask the LLM the question with all 10 facts in context. Note the answer.
Move the answer to fact #1. Ask again. Move it to fact #10. Ask again.
Compare the accuracy across positions.

For deeper exploration:

Read the original paper: search for "Lost in the Middle: How Language Models Use Long Contexts" by Liu et al.
Check out the MIT follow-up from 2025 that explains the causal masking mechanism — search for "Unpacking the bias of large language models MIT"
Search for "Found in the Middle calibration" — this paper proposes a calibration method that reduces position bias without retraining
Explore Ms-PoE (Multi-scale Positional Encoding) — a plug-and-play approach that improves middle-context utilization

Your RAG system retrieved five perfect chunks. The answer was in chunk #3. The LLM read chunk #1 carefully, skimmed chunks #2 through #4, and paid close attention to chunk #5. It's not carelessness — it's architecture. Causal masking and positional encodings create a structural blind spot in the middle. Once you know it's there, you can design around it: reorder your documents, slim down your context, and stop trusting that more tokens always means better answers.

Author: thousandmiles-ai-admin

LoRA and QLoRA Explained — Fine-Tune LLMs Without Selling Your Kidney for GPUs

Thousand Miles AI — Fri, 06 Mar 2026 05:31:58 +0000

Full fine-tuning a 7B model needs 4x A100 GPUs. You have a free Colab notebook with 15GB of RAM. Game over? Not even close. LoRA and QLoRA let you fine-tune billion-parameter models on hardware you already have. Here's how they actually work.

import Mermaid from "@/components/ui/mermaid"

The Problem We All Face

Imagine this: You just found the perfect dataset to fine-tune an LLM. Something domain-specific. Something that would make your startup, research project, or college assignment actually work. You Google "how to fine-tune Llama 2 7B" with excitement.

Five minutes later, you're staring at this:

"You'll need approximately 100-120 GB of VRAM. An A100 GPU costs $2-3 per hour. Full fine-tuning takes 20 hours minimum."

You check your resources. Free Google Colab. 15GB RAM. T4 GPU.

Your dreams are crushed. Or are they?

Enter LoRA and QLoRA — the techniques that say "nope, not today" to expensive GPU clusters and let you fine-tune GPT-scale models on hardware you already have.

Why Fine-Tuning Actually Matters

Before we dive into the magic, let's talk about why fine-tuning is worth the trouble.

Pre-trained LLMs are generalists. They're good at everything because they learned from everything. But "good at everything" often means "perfect for nothing." If you want an LLM that:

Writes technical documentation in your specific code style
Understands domain-specific jargon in medical, legal, or financial contexts
Responds in a particular tone or personality
Handles edge cases unique to your problem

...you need fine-tuning. It's the bridge between "generic chatbot" and "actually useful for my specific task."

But full fine-tuning? That's expensive. Really expensive.

What's Wrong with Full Fine-Tuning?

During full fine-tuning, you update every single weight in the model. For a 7-billion parameter model, that's:

7 billion trainable parameters
Each parameter needs gradients stored during backpropagation
GPU memory = (model size) + (gradients) + (optimizer states) + (batch)
Result: ~100-120 GB of VRAM needed

A single A100 GPU (the workhorse of AI) costs $2-3 per hour on cloud platforms. To fine-tune a 7B model, you'd need 4 of them, or find a different way.

This is where the magic happens.

The Insight: Models Are Secretly Low-Rank

Here's the key insight that changed everything: When you fine-tune a model on a new task, the weight updates don't require the full dimensionality of the original weights. Most of the "important" changes can be captured in a low-rank structure.

What does that mean?

Imagine a weight matrix W that's 4096 × 4096 (typical in transformer layers). That's ~16 million individual parameters. But the researchers behind LoRA discovered something: you don't need to update all 16 million parameters. You can approximate the weight updates using two much smaller matrices.

Let's visualize this:

Instead of updating 16.7 million parameters, LoRA updates only ~65,000 parameters (the two small matrices). That's 99.6% fewer parameters to train.

The magic formula is simple:

W_new = W_original + ΔW
ΔW ≈ A × B  (where A is 4096 × r, B is r × 4)

Here, r is the rank — a small hyperparameter you choose (typically 8-64). The lower the rank, the fewer parameters you train.

How LoRA Actually Works

Let's break down the LoRA training process step by step.

Step 1: Freeze the Base Model

Your pre-trained model stays completely frozen. Its 7 billion parameters don't change. This is huge for memory savings.

Step 2: Add Tiny Adapter Matrices

For every weight matrix you want to adapt (typically in the attention layers), you add two small matrices:

Matrix A: initialized randomly and small (e.g., 4096 × 8)
Matrix B: initialized to zero (this is important!)

Step 3: Train Only the Adapters

During fine-tuning, you only update A and B. These are trained using standard backpropagation on your downstream task.

Step 4: Merge During Inference (Optional)

After training, you can merge A and B into the original weights: W_new = W_original + A × B. This takes a few seconds and gives you a single model file with zero inference overhead.

Why this works:

Pre-trained weights = vast knowledge from billions of examples
LoRA adapters = task-specific knowledge from your dataset
Combined = best of both worlds

Here's the training flow:

Enter QLoRA: The Final Boss of Efficiency

LoRA is already mind-blowingly efficient. But what if you could go further?

QLoRA combines LoRA with 4-bit quantization — a technique that compresses the model weights to use only 4 bits per parameter instead of 32 bits (8x compression).

What is 4-Bit Quantization?

Instead of storing weights as full 32-bit floating-point numbers, you store them as 4-bit integers. Quantization is lossy (you lose some precision), but the loss is tiny.

Quantization formula:

4-bit weight ≈ original 32-bit weight
Compression: 32 bits → 4 bits = 8x smaller

QLoRA's Secret Sauce

QLoRA doesn't just quantize. It uses three clever tricks:

1. NF4 (Normalfloat) Data Type

A new data type mathematically optimal for normally distributed weights (which transformer weights are)
Better precision than standard 4-bit quantization
Information-theoretically superior

2. Double Quantization

Quantizes the quantization constants themselves
Reduces memory overhead further
Example: Instead of storing a 32-bit scaling factor per block, store a quantized version

3. Paged Optimizers

Manages memory spikes during backpropagation
Moves data to CPU RAM when GPU RAM is full
No crashes, just slower (but still fast)

The Numbers

Here's where it gets ridiculous:

Model	Full Fine-Tuning	LoRA	QLoRA
7B	100-120 GB	20-30 GB	8-12 GB
13B	200+ GB	40-50 GB	12-16 GB
70B	2000+ GB	400 GB	60-80 GB

QLoRA lets you fine-tune a 70B model on a single A100 80GB GPU. Full fine-tuning would need 4-8 of them.

Or, more relevant to you: fine-tune 7-13B models on a free Google Colab T4 GPU with 15GB RAM.

Actually Doing It: The Colab Path

This is where theory meets reality.

What You'll Need

Google Colab (free tier)
A Hugging Face account (free)
A small dataset (1,000-10,000 examples minimum)
The transformers, peft, and bitsandbytes libraries

High-Level Steps

Step 1: Load a Quantized Model

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_name = "mistralai/Mistral-7B-v0.1"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    load_in_4bit=True,  # QLoRA magic
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

Step 2: Add LoRA Adapters

from peft import LoraConfig, get_peft_model

lora_config = LoraConfig(
    r=8,  # rank
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],  # which weights to adapt
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

Step 3: Train

from transformers import Trainer, TrainingArguments

training_args = TrainingArguments(
    output_dir="./output",
    num_train_epochs=3,
    per_device_train_batch_size=4,  # tiny because of RAM
    gradient_accumulation_steps=4,
    save_steps=100,
    logging_steps=10
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset
)

trainer.train()

That's it. Seriously. The libraries handle the memory management, quantization, and LoRA logic for you.

The Practical Gotchas (Learn From My Pain)

Gotcha 1: Rank Selection is Non-Obvious

Lower rank = fewer parameters = faster, less memory.
Higher rank = more capacity = potentially better quality.

Wrong approach: Pick r=8 because it's small.

Right approach: Try r=8, 16, 32 and compare. For most 7B models, r=16-32 works well. For very small datasets, r=8 might even be better (less overfitting).

Gotcha 2: Small Datasets Overfit Easily

LoRA adapters are tiny — only ~1-5% of parameters. This is great for efficiency but terrible if your dataset has only 100 examples.

Wrong approach: "More epochs = better results"

Right approach: Start with 1 epoch, monitor validation loss. If validation loss increases while training loss decreases, you're overfitting. Add dropout, reduce rank, or get more data.

Gotcha 3: Forgetting to Merge

If you train LoRA adapters and then share the fine-tuned model, you have two options:

Keep A and B separate — saves ~1-5 MB, but requires the PEFT library to load
Merge into base weights — one large file, but works with standard transformers library

# Merge adapters into base model
merged_model = model.merge_and_unload()
merged_model.save_pretrained("./final_model")

Users downstream will appreciate the merged version. Don't forget this step.

Gotcha 4: Quantization Loss is Real

QLoRA is amazing, but 4-bit quantization does lose some information. For most tasks, it's negligible (80-90% of full fine-tuning quality). But for tasks requiring high precision (e.g., mathematical reasoning), consider LoRA without quantization if you have the RAM.

Next Steps: Actually Try This

Get a dataset — find one on Hugging Face Hub, or use a simple one like databricks-dolly-15k
Clone a Colab notebook — start with Hugging Face's QLoRA example
Modify for your task — change the model, dataset, and LoRA rank
Train — hit play and wait
Test — load your fine-tuned model and see if it actually works

The Bigger Picture

LoRA and QLoRA democratized fine-tuning. Five years ago, only teams with GPU budgets could customize LLMs. Now, any college student with a laptop and Colab access can do it.

These techniques are part of a broader movement toward Parameter-Efficient Fine-Tuning (PEFT) — methods that adapt massive models using tiny tweaks. LoRA and QLoRA aren't the only ones (there are prefix tuning, adapters, bitfit), but they're the most practical.

And they work. Teams fine-tune production models with LoRA every single day. It's not a research curiosity anymore — it's a standard tool.

Key Takeaways

Full fine-tuning is expensive. Updating 7 billion parameters requires 100+ GB GPU RAM.
LoRA trains 0.4% of parameters. It approximates weight updates using two small matrices instead of updating the whole model.
QLoRA adds 4-bit quantization. This lets you fine-tune 70B models on a single A100, or 7B models on free Colab.
The libraries do the heavy lifting. You don't need to implement LoRA — Hugging Face PEFT handles it.
Start small, experiment, iterate. Find the right rank, dataset size, and hyperparameters for your problem.

The GPU poverty problem isn't solved completely — but it's been heavily negotiated down. And that changes everything.

References

If you want to dig deeper, these resources are essential:

Happy fine-tuning. You've got this.

Author: thousandmiles-ai-admin

Chunking Strategies That Actually Work — Why Your RAG App Retrieves Garbage

Thousand Miles AI — Fri, 06 Mar 2026 05:31:28 +0000

Fixed-size, recursive, semantic — everyone has an opinion on the 'best' chunking strategy. The 2026 benchmarks are in, and the results will surprise you. Here's what actually works and why.

Chunking Strategies That Actually Work — Why Your RAG App Retrieves Garbage

The most boring part of your RAG pipeline is also the most consequential. Get chunking wrong, and nothing downstream can save you.

The Contract That Lied

Picture this. You're building a legal document assistant. A lawyer asks: "Is the company liable for damages in cases of force majeure?" Your RAG system retrieves a chunk that confidently states: "The company is liable for all damages arising from service interruptions." Clear answer, right?

Except... the original document said: "The company is liable for all damages arising from service interruptions, except in cases of force majeure as defined in Section 12." Your chunker, set to split every 500 tokens, sliced the sentence right between "interruptions" and "except." The exception — the most important part — ended up in the next chunk. That chunk didn't get retrieved because the query was about liability, not about Section 12 definitions.

One bad split. A completely wrong answer. And the user has no idea because the retrieved chunk looked perfectly valid.

This isn't a contrived example. This pattern plays out constantly in production RAG systems. Tables split in half. Lists separated from their headers. Paragraphs that say "as mentioned above" — but "above" is in a different chunk that wasn't retrieved. Chunking errors are silent, invisible, and devastating.

Why Should You Care?

Chunking is the first real decision in any RAG pipeline, and it's the one most developers spend the least time on. Everyone obsesses over embedding models, vector databases, and LLM selection — but the quality of your chunks puts a hard ceiling on everything else. You can't retrieve what you've destroyed.

And here's what makes it interesting: the 2026 benchmarks flipped a lot of assumptions on their head. The fanciest chunking methods? They're not winning. Understanding why requires understanding what each strategy actually does — and that's knowledge that shows up in both system design interviews and production debugging.

Let Me Back Up — What Is Chunking and Why Do We Need It?

Before your documents go into a vector database, they need to be broken into smaller pieces. There are two reasons for this.

First, embedding models have token limits. Most models cap out at 512 or 8,192 tokens. You can't embed a 50-page PDF as a single unit.

Second — and this is the less obvious reason — you want precision in retrieval. If your entire document is one big chunk, any query about any topic in that document will retrieve the whole thing. The LLM then has to find the needle in a haystack. Small, focused chunks mean the retriever can surface exactly the paragraph that answers the question.

But "small and focused" creates its own problem: the smaller the chunk, the more context it loses. A chunk that says "this approach" without telling you what "this" refers to is useless. The art of chunking is finding the sweet spot between precision and context.

The chunking step sits between your raw documents and the vector database. It determines the quality of everything downstream.

The Three Strategies — Explained Like You're Pair-Programming

Strategy 1: Fixed-Size Chunking

This is the "I don't want to think about it" approach. You pick a number — say 500 tokens — and split the text every 500 tokens. Done.

It's dead simple to implement, fast to run, and produces predictable chunk sizes. Most tutorials use this as the default, which is why most beginners start here.

The problem? It has zero awareness of your text's structure. It will split mid-sentence, mid-paragraph, mid-table. That liability clause we talked about? Fixed-size chunking is exactly how it gets destroyed.

The one saving grace is overlap. By repeating the last 50–100 tokens of each chunk at the beginning of the next one, you create a buffer zone where boundary information isn't completely lost. It's a patch, not a fix — but it helps more than you'd expect.

When it works: Flat, unstructured text like logs, transcripts, or scraped web content where there's no meaningful structure to preserve.

Strategy 2: Recursive Character Splitting

This is the strategy that actually won the 2026 benchmarks — and it's not even that complicated.

Instead of blindly splitting every N tokens, recursive splitting tries a hierarchy of separators. First, it tries to split on double newlines (paragraph breaks). If the resulting chunks are still too large, it splits on single newlines. Still too large? Sentences. Still too large? Words.

The key insight: it respects natural boundaries first and only gets more aggressive when it has to. A 500-token paragraph stays intact. A 2,000-token section gets split at paragraph boundaries, not mid-sentence.

Think of it like cutting a pizza. Fixed-size is a grid pattern — equal pieces but you cut through toppings. Recursive is cutting along the natural slice lines first, and only cutting slices in half if they're too big.

Most frameworks (LangChain, LlamaIndex) use this as their RecursiveCharacterTextSplitter. The default separator hierarchy is: ["\n\n", "\n", " ", ""] — paragraphs, lines, words, characters.

When it works: Almost everything. The 2026 FloTorch benchmark tested seven strategies across thousands of documents, and recursive splitting at 512 tokens achieved the highest answer accuracy and retrieval F1 scores.

Strategy 3: Semantic Chunking

Okay, here's where it gets interesting — and controversial.

Semantic chunking uses embeddings to determine where to split. It embeds each sentence individually, then measures the similarity between consecutive sentences. When the similarity drops below a threshold — meaning the topic changed — it places a split there.

The idea is elegant: instead of splitting based on character count, you split based on meaning. Each chunk should be a coherent unit about one topic.

The problem? It's expensive and surprisingly inconsistent. You need to generate embeddings for every sentence just to decide where to split. For a large corpus, that means thousands of API calls or significant local compute before you've even started indexing.

And the 2026 benchmarks showed something counterintuitive: semantic chunking often produced worse retrieval than recursive splitting. Why? Because semantic chunks vary wildly in size. Some end up with 50 tokens (too small for meaningful embedding), others with 2,000+ tokens (too large for precise retrieval). The inconsistent size makes it harder for the retriever to compare chunks fairly.

The same legal clause, chunked three ways. Fixed-size breaks it. Recursive preserves it. Semantic groups it by topic but may bundle too much.

When it works: Multi-topic narrative documents where topics shift unpredictably — research papers, long blog posts, interview transcripts. But only if you can afford the compute and are willing to tune the similarity threshold.

The 2026 Surprise — Why Simpler Is Winning

Here's what caught everyone off guard. When comprehensive benchmarks tested all these strategies head-to-head, the ranking was:

Recursive splitting (512 tokens) — highest accuracy, highest retrieval F1
Fixed-size (512 tokens with overlap) — close second
Semantic chunking — middle of the pack
Proposition-based chunking (using LLMs to decompose) — expensive, marginally better on some tasks

The reason simpler methods won isn't that they're inherently superior — it's that they produce consistent, predictable chunk sizes. Embedding models and retrievers are optimized for chunks in the 256–512 token range. When your chunks are consistently in that range, the entire pipeline works more predictably.

Semantic and proposition-based methods also create 3–5x more chunks for the same corpus. More chunks means more embeddings, more storage, more compute, and — counterintuitively — more noise in retrieval. The cost multiplier compounds at every layer.

Does the 3% accuracy improvement from semantic chunking justify 10x the processing cost? For most applications, no.

Mistakes That Bite — The Chunking Errors Nobody Talks About

"I'll just use the default 1,000-token chunks." This is the most common error. Most tutorials and framework defaults use 1,000 tokens. But most embedding models are optimized for 256–512 tokens. Larger chunks dilute the embedding — instead of representing one specific idea, they represent a fuzzy average of several ideas. Drop to 512 with 50-token overlap and measure the difference.

"Chunking is a one-time decision." Different document types need different strategies. Your API docs might work perfectly with recursive splitting, while your meeting transcripts might need semantic chunking. Don't apply one strategy to your entire corpus blindly. Profile your document types and test each one.

"Tables and code can be chunked like prose." They absolutely cannot. A table split in half is worse than useless — it's misleading. Code split mid-function is syntactically invalid. Extract tables and code blocks as separate units, preserve their structure, and add surrounding context (the header before the table, the function name, the paragraph that references it).

Now Go Break Something — Where to Go from Here

Here's a weekend experiment that'll teach you more about chunking than any article:

Take a document you know well — your project docs, college notes, anything where you can verify the answers.
Chunk it three ways using LangChain's splitters: CharacterTextSplitter (fixed), RecursiveCharacterTextSplitter (recursive), and SemanticChunker from langchain-experimental.
Ask the same 10 questions to a RAG pipeline using each chunking strategy.
Compare the retrieved chunks side by side. You'll immediately see where fixed-size destroys context, where recursive preserves it, and where semantic produces inconsistent sizes.

For deeper exploration:

Search for "FloTorch RAG chunking benchmark 2026" — the full benchmark results with methodology
The LangChain docs have a great comparison page for all their text splitters
Check out the Weaviate blog's chunking guide — it has practical examples for different document types
For advanced work, look into "late chunking" — a newer approach where you embed the full document first and then chunk the embeddings, preserving long-range context

That legal document assistant that said "the company is liable for all damages"? It wasn't lying — it was reading the only chunk it had, and that chunk had been amputated mid-sentence. Swap to recursive splitting at 512 tokens with overlap, and the full clause — exception included — stays intact. The fix wasn't a better model or a smarter prompt. It was a better cut.

Author: Shibin

Build Your First MCP Server in Python — A Weekend Project That Actually Impresses

Thousand Miles AI — Fri, 06 Mar 2026 05:31:24 +0000

You've heard MCP is the 'USB-C for AI.' But what does it take to actually build one? A hands-on walkthrough of creating an MCP server from scratch using Python and FastMCP — with tools your LLM can call.

Build Your First MCP Server in Python — A Weekend Project That Actually Impresses

Everyone talks about MCP. Very few people have actually built a server. Here's how to be one of them — in about an hour.

The Moment Your LLM Gets Hands

You've been playing with ChatGPT or Claude for months. Asking questions, generating code, summarizing documents. But there's always a wall: the model can only work with what you give it. Want it to check today's weather? You copy-paste from a browser. Want it to query your database? You run the query yourself and paste the results.

Now imagine you told your AI: "What's the weather in Chennai right now?" — and it actually went and fetched the answer. Not from training data. Not from a cached response. It called a real API, got real-time data, and gave you the result.

That's what an MCP server lets you do. You build a small Python service that exposes "tools" — functions your LLM can discover and call. The LLM sees what tools are available, decides which one to use, passes the right parameters, and gets the response back. No copy-pasting. No manual plumbing.

And the best part? The server you build works with any MCP-compatible client — Claude Desktop, Cursor, VS Code, or any custom app.

Why Should You Care?

Two reasons. First, MCP is becoming the standard way AI agents interact with external tools, adopted by both Anthropic and OpenAI, and governed by the Linux Foundation. Knowing how to build MCP servers is a genuinely useful skill for any AI-focused role.

Second — and more practically — this is one of the best portfolio projects you can build right now. Most people's AI projects are "I wrapped an API call in a chatbot." Building an MCP server shows you understand protocols, tool design, and how AI agents actually connect to the real world. That's a very different conversation in an interview.

Let Me Back Up — What Are We Building?

We're going to build a Python MCP server that exposes three tools to any LLM client:

get_weather — Fetches current weather for any city using a free API
calculate — Evaluates a math expression safely
random_fact — Returns a random fun fact (because why not)

The server will run locally on your machine. We'll connect it to Claude Desktop so you can actually chat with an AI that uses your tools. The whole thing takes about 50 lines of Python.

What we're building: a Python server that exposes tools to Claude Desktop via MCP.

Okay, Let's Build It — Step by Step

Step 1: Set Up Your Environment

You need Python 3.10 or higher. If you're on a Mac or Linux machine, you probably already have it. Check with python3 --version.

Create a project folder and set up a virtual environment:

mkdir my-mcp-server && cd my-mcp-server
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

Install the MCP SDK. The recommended way is using pip:

pip install "mcp[cli]" httpx

mcp is the official MCP Python SDK. httpx is for making HTTP requests to external APIs. We're also pulling in FastMCP, which is included in the SDK and gives us a clean decorator-based API for defining tools.

Step 2: Write the Server

Create a file called server.py. Here's the entire thing:

from mcp.server.fastmcp import FastMCP
import httpx
import random

# Create the MCP server
mcp = FastMCP("my-first-server")

# Tool 1: Get weather for a city
@mcp.tool()
async def get_weather(city: str) -> str:
    """Get current weather for a city. Returns temperature and conditions."""
    async with httpx.AsyncClient() as client:
        response = await client.get(
            f"https://wttr.in/{city}?format=j1"
        )
        data = response.json()
        current = data["current_condition"][0]
        temp = current["temp_C"]
        desc = current["weatherDesc"][0]["value"]
        return f"{city}: {temp}°C, {desc}"

# Tool 2: Safe calculator
@mcp.tool()
async def calculate(expression: str) -> str:
    """Evaluate a math expression safely. Example: '2 + 3 * 4'"""
    allowed = set("0123456789+-*/.() ")
    if not all(c in allowed for c in expression):
        return "Error: Only numbers and basic operators allowed"
    try:
        result = eval(expression)  # Safe because we filtered input
        return f"{expression} = {result}"
    except Exception as e:
        return f"Error: {str(e)}"

# Tool 3: Random fun fact
@mcp.tool()
async def random_fact() -> str:
    """Return a random fun fact about technology or science."""
    facts = [
        "The first computer bug was an actual bug — a moth found in a Harvard Mark II computer in 1947.",
        "The first 1GB hard drive, introduced in 1980, weighed about 250 kg and cost $40,000.",
        "About 90% of the world's data was created in the last two years.",
        "The average smartphone today has more computing power than NASA had for the Apollo 11 moon landing.",
        "The first website ever created is still online at info.cern.ch.",
    ]
    return random.choice(facts)

if __name__ == "__main__":
    mcp.run(transport="stdio")

That's it. Seriously. Let's break down what's happening.

Step 3: Understand What You Just Wrote

FastMCP is a wrapper from the official SDK that makes defining tools dead simple. You create an instance, decorate your functions with @mcp.tool(), and FastMCP handles all the MCP protocol stuff — JSON-RPC messages, tool discovery, parameter validation.

Each tool function has:

Type hints — city: str tells the LLM what parameters the tool expects
A docstring — This is critical. The LLM reads this to decide when to use the tool. Write it like you're explaining the tool to a person.
A return value — A string that gets sent back to the LLM as the observation

The mcp.run(transport="stdio") at the bottom starts the server using standard input/output — this is how Claude Desktop communicates with local MCP servers. No HTTP, no ports, just stdin/stdout.

The full flow: you ask a question, Claude decides to use your tool, the server calls the API, and the result flows back.

Step 4: Connect to Claude Desktop

Now the fun part — making Claude Desktop actually use your server.

Open Claude Desktop and go to Settings > Developer > Edit Config. This opens a JSON file. Add your server to it:

{
  "mcpServers": {
    "my-first-server": {
      "command": "python3",
      "args": ["/full/path/to/your/server.py"],
      "env": {}
    }
  }
}

Replace /full/path/to/your/server.py with the actual path to your file. Save, and restart Claude Desktop.

You should now see a small hammer icon in the chat input area — that means Claude has discovered your tools. Click it to see the three tools listed.

Step 5: Test It

Type into Claude: "What's the weather in Bangalore right now?"

Claude should recognize that it needs to use the get_weather tool, call your server, and return the live weather data. Try the calculator: "What's 15 * 37 + 42?" Try the fun fact: "Tell me a random tech fact."

Each time, you'll see Claude decide which tool to use, call it through your MCP server, and incorporate the result into its response. You've just given an LLM the ability to do things it couldn't do before.

Making It Better — Ideas for Your Next Steps

The basic server works, but here are a few directions to take it further:

Add more tools. Wrap any API you use regularly — a to-do list API, a movie database, your college's timetable system, a GitHub API for checking your repos. Each tool is just another decorated function.

Add resources. Tools let the LLM do things. Resources let it read things. You can expose file contents, database records, or API responses as read-only resources that the LLM can pull into its context.

Try the inspector. The MCP SDK ships with a built-in inspector tool. Run mcp dev server.py to get a web UI where you can test your tools interactively without needing Claude Desktop. Super useful for debugging.

Deploy it remotely. Local servers are great for development, but for production you'd want an HTTP-based transport (SSE or WebSocket). The FastMCP docs cover this — it's a one-line change from stdio to sse.

Mistakes That Bite — Things That Trip Up Beginners

"My tools don't show up in Claude Desktop." The most common issue. Double-check: is the path in the config JSON absolute? Is the virtual environment activated? Did you restart Claude Desktop after editing the config? The server needs to start successfully for tools to appear.

"The docstring doesn't matter, right?" Wrong. The LLM uses the docstring to decide whether and when to use your tool. A vague docstring like "does something with weather" will confuse the model. Be specific: "Get current weather for a city. Returns temperature in Celsius and conditions."

"I'll just expose my entire database as a tool." Resist this urge. Each tool should do one specific thing. A tool called query_anything that accepts raw SQL is both a security nightmare and confusing for the LLM. Instead, create focused tools like get_user_by_email or list_recent_orders. Smaller, focused tools get used correctly more often.

Now Go Break Something

You've just built something that most developers only read about. An MCP server — the standard protocol that major AI labs are converging on — running on your machine, giving an LLM real-world capabilities.

Here's what to explore next:

The official MCP docs at modelcontextprotocol.io have a quickstart guide and full API reference
The free Hugging Face MCP course walks through building servers and connecting them to agents, with hands-on exercises
FastMCP's GitHub repo has examples for advanced patterns like authentication, streaming, and resource management
Search for "MCP server examples" on GitHub — the community has built servers for Notion, Kubernetes, Spotify, and hundreds of other services. Reading other people's servers is one of the fastest ways to learn

Remember staring at your LLM, wishing it could just check the weather or run a calculation instead of making you do it? Fifty lines of Python later, it can. That's what MCP is about — not replacing what LLMs do well, but giving them the tools to do what they couldn't. Your server is small. The pattern scales to anything.

Author: Shibin