DEV Community: VLAD

Why more context makes your AI answers worse

VLAD — Mon, 31 Aug 2026 22:16:08 +0000

Your model says it has a one-million-token context window. Its real working memory is a lot smaller than that.

On long-context benchmarks, models start failing well below the number printed on the box. And here's the part nobody warns you about: past a certain point, adding more context makes your answers worse. Not better. Worse.

Prefer to watch? Full walkthrough with the attention-cost animation:

A context window is not memory

Let's kill one idea first. A context window is not memory.

The model doesn't remember your last message — it's stateless. It keeps nothing between calls. Every single call, you send the whole conversation again. All of it.

Look at what you actually ship over the wire:

Turn one: a system prompt and one question. Small.
Turn ten: the same system prompt, plus nine questions, plus nine answers, plus the new one.

The window is just how much text fits in that box before the API says no. And everything in the box gets processed. Every time.

What attention actually does

Inside the model, attention has a job that's simple to say: every token looks at every other token and decides how much it cares about each one.

10 tokens → 100 pairs
1,000 tokens → 1,000,000 pairs
100,000 tokens → 10,000,000,000 pairs

Ten times the context isn't ten times the work. It's a hundred times the work. (That's just arithmetic: cost grows with the square of the input.)

That's also why the first reply in a long chat feels slow and the rest feel fast — the model caches the keys and values it already computed. But the cache saves you compute. It doesn't save your answer quality. That's a separate problem, and it's the one you actually feel.

Here are the three reasons big context hurts.

Reason 1 — attention is a budget

For every token, the attention weights add up to one. Always one — that's what softmax does.

So when you paste in fifty more documents, you don't hand the model more focus. You split the same focus into more pieces.

Think about search on your laptop. One folder, ten files — you find it in a second. Same search, ten thousand files, same query, and the right file is now somewhere on page four. The model has that exact problem. Except it never shows you page four. It just answers.

Reason 2 — lost in the middle

Researchers took one correct answer and moved it around inside a long prompt. Same prompt, same question, different position. Then they measured accuracy.

Near the beginning: good.
Near the end: good.
In the middle: much worse.

(Liu et al., Lost in the Middle.) So your most important paragraph, sitting at 60% of a long prompt, is in the worst possible spot in the whole file.

Reason 3 — distractors in your own data

This is the one I hit most in real code. The famous needle-in-a-haystack test is easy, because the needle looks nothing like the hay. Your production data is not like that.

You've got version two of the doc. Version three. An old changelog. And a chat thread that contradicts all three. Four chunks that all look right. One of them is right — and the model can't tell which.

It gets worse as the input grows: accuracy drops steadily with input length, even on tasks the same model nails at short length. So the number on the box tells you what fits. It doesn't tell you what works.

Four things you can actually change

1. Send less, but better. If you're doing retrieval, stop pushing the top fifty chunks into the prompt. Search wide, re-sort the results, then send the top three to five. Fewer clean chunks beat more chunks, every time.

2. Use the edges. Instruction at the top. Data in the middle. Then repeat the actual question at the bottom, right before the model answers. That last line costs you nothing, and it works.

3. Stop reusing one giant thread. Split the job into steps. Each step gets a fresh, small context with only what it needs. Between steps, pass a short summary — not the full transcript. That's most of what agent frameworks are doing for you under the hood.

4. Measure it on your own data. Take twenty real questions from your app. Run them with 2,000 tokens of context, then run the same twenty with 20,000. Compare the answers side by side. You'll find your own limit — and it'll be lower than the spec sheet.

The takeaway

The context window is a hard limit on what fits. It was never a promise about what the model can use. Treat it like bandwidth you're paying for, and send the smallest thing that answers the question.

What's the biggest prompt you've ever shipped to production? And when you cut it down, did it get better or worse? Tell me below.

I make Vlad's Stack — how the AI tools you use every day actually work, for people who write code. Full video walkthrough is above.

Why your RAG returns garbage (and it's not the model)

VLAD — Mon, 31 Aug 2026 22:12:03 +0000

Your RAG bot just gave a confident, detailed answer. And it's completely wrong.

Here's the part that'll annoy you: the model did nothing wrong. It answered perfectly — using the text you handed it. The bug isn't in the AI. It's in the five steps before the AI.

Prefer to watch? Full 6-minute walkthrough with the "lost in the middle" animation:

The pipeline, in one line of code

RAG is simple on paper. You search your own documents, grab the best matches, paste them into the prompt, and the model answers from those. So when the answer's garbage, everyone blames the model.

Wrong suspect. There are five places this pipeline breaks — and four of them happen before the model even runs:

const chunks  = split(docs)                   // 1 · chunking
const vectors = embed(chunks)                 // 2 · embedding
const hits    = search(embed(query), vectors) // 3 · retrieval
const best    = rerank(query, hits)           // 4 · ranking
const answer  = llm(prompt(best, query))      // 5 · generation

Five lines. Every bug I'm about to show you lives on one of them.

1. Chunking — a fixed cut splits your answer in half

Your documents are long, so you cut them into pieces before you store them. Most people cut every few hundred characters and move on. That's where it starts to rot.

A fixed-length cut doesn't care about meaning. It'll slice a sentence in half. It'll split a question from its answer. Now the one chunk that held your answer is two halves — and each half looks irrelevant on its own. So retrieval never even finds it.

Fix: cut on structure, not length. Split on paragraphs and headers — where the meaning actually breaks. And let chunks overlap a little, so nothing gets stranded at the edge.

2. Embedding — your question doesn't look like its answer

Every chunk goes through a model that turns text into a vector: a long list of numbers that captures its meaning. Similar meaning, similar numbers. That's the whole trick that makes search work.

But here's the trap. Your question gets embedded by the same model — and a question rarely looks like its answer.

You ask "how do I reset my password?" The doc says "account recovery procedure." Same thing to a human. Different words — and a weak embedder puts them far apart.

Fix: pick an embedding model built for retrieval, and test it on your data. A model that's great on legal text can be useless on code.

3. Retrieval — vector search fumbles exact strings

You take the question's vector and grab the closest chunks. This is the "vector search" everyone talks about. On its own, it's got a blind spot.

Vector search matches meaning, not exact words. Which is great — until someone searches for an error code. A product name. SKU-4417. Those barely have a meaning to embed. They're just exact strings, and nothing is "close in meaning" to a serial number.

Fix: hybrid search. Run keyword search and vector search, then merge the results. Old-school keyword matching catches the exact strings embeddings miss.

4. Ranking — the best chunk was #8, and you dropped it

Now you've got a pile of candidates — say the top twenty. But you can't paste twenty chunks into the prompt, so you keep the top five and drop the rest.

Here's the question nobody asks: what if the best chunk was ranked number eight? Then you just threw it away. Your answer was in the pile, and you cut it.

Fix: a re-ranker — a second, smarter model that reads the question and each chunk together and re-scores them. Cheap vector search casts a wide net; the re-ranker picks the real winners out of it. Now your top five are actually the top five.

And order matters too — "lost in the middle"

Even the order you paste chunks in changes the answer. Models pay the most attention to the start and the end of the context. Put your most important chunk in the middle, and the model can skim right past it.

This isn't a hunch — it's measured. The same model gets more accurate just by moving the right chunk to the edges of the context (Liu et al., Lost in the Middle, 2023). So put your best chunk first, or last. Never buried in the middle.

5. The prompt — where people ruin it two ways

Right chunks, right order. This is your last chance to ruin it, and people do it two ways:

They dump in everything. Ten thousand tokens of maybe-relevant text, hoping the model sorts it out. More context isn't more accuracy — it's more noise to get lost in.
They never tell the model what to do when the answer isn't there. So it does the one thing you don't want: it makes something up.

Fix is basically one line in your prompt:

"Answer only from the context below. If it's not there, say you don't know."

That single instruction turns a confident liar into a system you can trust. A RAG bot that admits "I don't know" beats ten that guess.

Put the blame where it belongs

Bad chunks
Wrong embedder
Vector-only search
No re-ranker
A sloppy prompt

Five failure points — and notice the model was the last one. Usually the innocent one.

So next time your RAG bot lies to you, don't touch the model first. Walk the five steps backward — prompt, ranking, retrieval, embedding, chunks. The bug is almost always upstream.

What's the worst answer your RAG bot has ever given you? Drop it in the comments — I actually want to read those.

I make Vlad's Stack https://www.youtube.com/channel/UCUO8Uo5LsEy1b9eRkrH6JNg — how the AI tools you use every day actually work, for people who write code. Full video walkthrough of this one is above.

What a Linux container really is (it's not a VM)

VLAD — Mon, 31 Aug 2026 22:10:38 +0000

Right now, on your machine, a process is being lied to.

It thinks it's alone on the computer. It thinks it owns the whole filesystem. It thinks its process ID is 1. None of that is true. It's running right next to everything else, on the same kernel — and the kernel is lying to its face.

That lie has a name. We call it a container.

Prefer to watch? Full 6-minute walkthrough with the animation:

There is no "container" in the kernel

Here's the first thing nobody tells you. Open the Linux kernel source, search for a thing called a "container." You'll find zero. There is no container object. The kernel has no idea what the word means.

So what are you actually running? A normal process. The same kind as your text editor. Just with a few settings flipped. Docker doesn't build a magic box around your app — it starts a plain process and flips those settings for you.

There are two settings that do all the work:

The first controls what the process can see.
The second controls what it can use.

That's the whole magic. See, and use.

Trick 1 — namespaces (what a process can see)

Normally every process on the machine shares one view of the world: the same list of process IDs, the same filesystem, the same network. A namespace hands one process its own private copy of that view.

Put it in a PID namespace, and it sees itself as process 1 — like the machine just booted for it.
Put it in a mount namespace, it gets its own filesystem root.
Give it a network namespace, and it has its own network interface and its own IP.

So on the real host, your process might be number 842, sitting in a list of hundreds. Inside its namespace, it looks down and sees... 1. Same machine, same kernel. The process just can't see past the wall the kernel drew around it.

root inside, nobody outside

Here's the detail that trips everyone up: there's a user namespace too.

Inside the container, your process can be root — user zero, full power. Outside, on the host, that same process is a boring, unprivileged user.

Root on the inside. Nobody on the outside. Same process. That's namespaces doing their job — and it's the foundation of rootless containers.

Trick 2 — cgroups (what a process can use)

Namespaces decide what you see. The second trick decides what you get. It's called a cgroup — a control group.

A cgroup sets limits. This process gets two CPU cores, not all of them. Half a gig of memory, and not a byte more. Cross the memory limit? The kernel kills it on the spot (that's your OOM kill).

That's how one container can't starve every other one on the box. No limits, and one noisy app eats the whole machine.

The whole idea in one line

Namespaces are what you see. Cgroups are what you get.

A container is just a normal process wrapped in those two things. Nothing else is in the box — because there is no box.

Why it's not a virtual machine

This is the part people get wrong the most.

A virtual machine boots its own kernel, on fake hardware that a hypervisor pretends is real. A whole operating system, from scratch, every time.

A container has no kernel of its own. It borrows the host's kernel — the one that's already running.

That one difference explains everything:

Speed. No kernel to boot means a container starts in milliseconds, where a VM takes seconds.
Isolation. Sharing a kernel cuts both ways. One kernel bug can let a process break out of its container and onto the host. A VM doesn't share that wall.

So you're trading isolation for speed. That's the deal — and it's a fair one, as long as you know you're making it.

Build one by hand — no Docker

Don't believe there's no magic? Build a "container" yourself with two commands:

# a "container" with no Docker at all
unshare --pid --mount --net --uts --fork \
  chroot ./rootfs \
  /bin/sh

unshare flips on the namespaces.
chroot swaps the root filesystem.
And you're dropped into a shell that thinks it's alone on a fresh machine.

No daemon, no image — just kernel features.

So what does `docker run` actually do?

Four steps, in pseudo-code:

unpackImage(image)        // files become the new root
applyNamespaces(process)  // its own PID, mounts, network
applyCgroups(process)     // capped CPU and memory
exec(process)             // now it's just... a process

Unpack the files. Wrap it in namespaces. Cap it with cgroups. Run it. That's the whole thing Docker does that felt like magic.

And the image everyone ships around? It's just a stack of files in a tarball, plus a little metadata saying which program to start. No operating system inside. No kernel inside. Just files.

One number to leave you with

The Linux manual page namespaces(7) lists eight separate kinds of namespace — eight different views the kernel can hand a single process: mount, PID, network, IPC, UTS, user, cgroup, and time. That's the whole toolbox a container is built from.

So next time someone draws a container as a little box floating above the machine — remember there's no box. There's your kernel, showing one process a smaller version of the truth. Namespaces for what it sees, cgroups for what it gets, one shared kernel underneath.

What did you think a container was, before this? Tell me in the comments.

I make Vlad's Stack https://www.youtube.com/channel/UCUO8Uo5LsEy1b9eRkrH6JNg — how the tools you use every day actually work, for people who write code. Full video walkthrough of this one is above.

Why more context makes your AI answers worse

VLAD — Mon, 31 Aug 2026 22:09:07 +0000

Your model says it has a one-million-token context window. Its real working memory is a lot smaller than that.

Prefer to watch? Full walkthrough with the attention-cost animation:

A context window is not memory

Let's kill one idea first. A context window is not memory.

The model doesn't remember your last message — it's stateless. It keeps nothing between calls. Every single call, you send the whole conversation again. All of it.

Look at what you actually ship over the wire:

Turn one: a system prompt and one question. Small.
Turn ten: the same system prompt, plus nine questions, plus nine answers, plus the new one.

The window is just how much text fits in that box before the API says no. And everything in the box gets processed. Every time.

What attention actually does

Inside the model, attention has a job that's simple to say: every token looks at every other token and decides how much it cares about each one.

10 tokens → 100 pairs
1,000 tokens → 1,000,000 pairs
100,000 tokens → 10,000,000,000 pairs

Ten times the context isn't ten times the work. It's a hundred times the work. (That's just arithmetic: cost grows with the square of the input.)

Here are the three reasons big context hurts.

Reason 1 — attention is a budget

For every token, the attention weights add up to one. Always one — that's what softmax does.

So when you paste in fifty more documents, you don't hand the model more focus. You split the same focus into more pieces.

Reason 2 — lost in the middle

Researchers took one correct answer and moved it around inside a long prompt. Same prompt, same question, different position. Then they measured accuracy.

Near the beginning: good.
Near the end: good.
In the middle: much worse.

(Liu et al., Lost in the Middle.) So your most important paragraph, sitting at 60% of a long prompt, is in the worst possible spot in the whole file.

Reason 3 — distractors in your own data

This is the one I hit most in real code. The famous needle-in-a-haystack test is easy, because the needle looks nothing like the hay. Your production data is not like that.

Four things you can actually change

2. Use the edges. Instruction at the top. Data in the middle. Then repeat the actual question at the bottom, right before the model answers. That last line costs you nothing, and it works.

The takeaway

The context window is a hard limit on what fits. It was never a promise about what the model can use. Treat it like bandwidth you're paying for, and send the smallest thing that answers the question.

What's the biggest prompt you've ever shipped to production? And when you cut it down, did it get better or worse? Tell me below.

I make Vlad's Stack https://www.youtube.com/channel/UCUO8Uo5LsEy1b9eRkrH6JNg — how the AI tools you use every day actually work, for people who write code. Full video walkthrough is above.

Why more context makes your AI answers worse

VLAD — Mon, 31 Aug 2026 21:42:30 +0000

Your model says it has a one-million-token context window. Its real working memory is a lot smaller than that.

Prefer to watch? Full walkthrough with the attention-cost animation:

A context window is not memory

Let's kill one idea first. A context window is not memory.

The model doesn't remember your last message — it's stateless. It keeps nothing between calls. Every single call, you send the whole conversation again. All of it.

Look at what you actually ship over the wire:

Turn one: a system prompt and one question. Small.
Turn ten: the same system prompt, plus nine questions, plus nine answers, plus the new one.

The window is just how much text fits in that box before the API says no. And everything in the box gets processed. Every time.

What attention actually does

Inside the model, attention has a job that's simple to say: every token looks at every other token and decides how much it cares about each one.

10 tokens → 100 pairs
1,000 tokens → 1,000,000 pairs
100,000 tokens → 10,000,000,000 pairs

Ten times the context isn't ten times the work. It's a hundred times the work. (That's just arithmetic: cost grows with the square of the input.)

Here are the three reasons big context hurts.

Reason 1 — attention is a budget

For every token, the attention weights add up to one. Always one — that's what softmax does.

So when you paste in fifty more documents, you don't hand the model more focus. You split the same focus into more pieces.

Reason 2 — lost in the middle

Researchers took one correct answer and moved it around inside a long prompt. Same prompt, same question, different position. Then they measured accuracy.

Near the beginning: good.
Near the end: good.
In the middle: much worse.

(Liu et al., Lost in the Middle.) So your most important paragraph, sitting at 60% of a long prompt, is in the worst possible spot in the whole file.

Reason 3 — distractors in your own data

This is the one I hit most in real code. The famous needle-in-a-haystack test is easy, because the needle looks nothing like the hay. Your production data is not like that.

Four things you can actually change

2. Use the edges. Instruction at the top. Data in the middle. Then repeat the actual question at the bottom, right before the model answers. That last line costs you nothing, and it works.

The takeaway

The context window is a hard limit on what fits. It was never a promise about what the model can use. Treat it like bandwidth you're paying for, and send the smallest thing that answers the question.

What's the biggest prompt you've ever shipped to production? And when you cut it down, did it get better or worse? Tell me below.

DEV Community: VLAD

Why more context makes your AI answers worse

A context window is not memory

What attention actually does

Reason 1 — attention is a budget

Reason 2 — lost in the middle

Reason 3 — distractors in your own data

Four things you can actually change

The takeaway

Why your RAG returns garbage (and it's not the model)

The pipeline, in one line of code

1. Chunking — a fixed cut splits your answer in half

2. Embedding — your question doesn't look like its answer

3. Retrieval — vector search fumbles exact strings

4. Ranking — the best chunk was #8, and you dropped it

And order matters too — "lost in the middle"

5. The prompt — where people ruin it two ways

Put the blame where it belongs

What a Linux container really is (it's not a VM)

There is no "container" in the kernel

Trick 1 — namespaces (what a process can see)

root inside, nobody outside

Trick 2 — cgroups (what a process can use)

The whole idea in one line

Why it's not a virtual machine

Build one by hand — no Docker

So what does docker run actually do?

One number to leave you with

Why more context makes your AI answers worse

A context window is not memory

What attention actually does

Reason 1 — attention is a budget

Reason 2 — lost in the middle

Reason 3 — distractors in your own data

Four things you can actually change

The takeaway

Why more context makes your AI answers worse

A context window is not memory

What attention actually does

Reason 1 — attention is a budget

Reason 2 — lost in the middle

Reason 3 — distractors in your own data

Four things you can actually change

The takeaway

So what does `docker run` actually do?