DEV Community: AI Engineering Illustrated

How Cursor Got Thousands of AI Agents to Build One Browser

AI Engineering Illustrated — Wed, 14 Oct 2026 10:45:27 +0000

Locks failed, one boss agent burned out, and a tree of planners won. Here's what Cursor's multi-agent experiments teach about coordinating agents at scale.

One coding agent is good at a focused task: fix this bug, add this endpoint, write these tests. Give it something the size of a web browser and it falls apart. Cursor engineer Wilson Lin tried exactly that. He asked a frontier model to plan a browser engine and kept telling it to "keep going." The model lost track of what it was doing, declared success long before it was finished, and got stuck on hard details.

So Cursor asked a different question: if you spend 10× more compute, can you get 10× more useful work? In a research project that ran for a few weeks around the start of 2026, the team built a harness that ran hundreds of agents per machine, about 2,000 at peak, on a single codebase. It wrote a browser engine called FastRender with more than a million lines of Rust. At its peak, the system landed about 1,000 commits an hour with no human steering.

The browser was only the test case. The real output was a set of coordination lessons that became a product: Cursor Projects, launched September 10, 2026, where one coordinator agent directs thousands of subagents. This issue walks through the four designs Cursor tried, why three failed, how the winner works, and what it cost.

Background: Why One Agent Isn't Enough

A few terms first.

Agent: a language model calling tools in a loop: read files, run commands, edit code, repeat.
Harness: the program around the model that runs that loop, supplies tools and manages context. (Issue 005 took a single-agent harness apart.)
Multi-agent system: several agents, each with its own context window, splitting up one goal.
Throughput: useful work per hour. For Cursor, mostly commits that move the project forward.

A single agent has two hard limits on a big project. It does one thing at a time, and it has one context window that fills with old tool output until the agent loses the plot. Many agents, each with a fresh context, fix both in principle. In practice they create the problem every human team has: coordination. Who works on what? How do you stop two agents fixing the same bug? Who decides the project is done?

Cursor's research is useful because it reports the designs that didn't work. Lin describes it in two posts: Scaling long-running autonomous coding (January 14, 2026) and the longer Towards self-driving codebases (February 5, 2026). Everything below comes from those posts unless another source is linked.

Four Harness Designs, Three Failures

The team started with a simple Rust harness on one large Linux VM, controlled over SSH. Before tuning anything, they logged every agent message, system action and command output with timestamps, so sessions could be replayed and mined for patterns. That observability made every later step possible.

1. Flat peers and a lock file. The first idea was the least prescriptive: every agent has the same role, and they coordinate through a shared state file. An agent reads the file to see what others are doing, claims a task and updates its status, with a lock so two agents can't grab the same task.

It failed fast. Agents held locks too long, forgot to release them, or edited the file without locking at all. Even when locking worked, 20 agents slowed to the throughput of 2 or 3, mostly waiting. Swapping locks for optimistic concurrency control (read freely; a write fails if the state changed since you read it) was more robust, but exposed a deeper problem. With no hierarchy, agents became risk-averse, picking small, safe changes. No one owned the project, so work churned without progress.

2. Planner, executor, workers and a judge. Next, roles. A planner wrote the approach and deliverables. An executor owned delivering that plan and spawned workers to do the tasks. After each round, an independent judge decided whether the work was finished or another iteration should run. This fixed most coordination problems, because a single agent now owned the outcome. But it was rigid. Each round waited for the slowest worker, and because all planning happened up front, the system couldn't adjust when it discovered new problems mid-round.

3. The continuous executor. So Cursor removed the separate planner. The executor planned and spawned tasks continuously, and to keep it from drifting over long runs the team added "freshness" mechanisms:

a scratchpad.md that agents rewrite rather than append to,
automatic summarizing when an agent nears its context limit,
self-reflection and alignment reminders in the system prompts,
explicit permission to pivot and challenge assumptions.

It worked well enough that the judge was dropped. Then the executor started misbehaving: sleeping at random, doing work itself, spawning only a few narrow tasks, merging badly and claiming early completion. The diagnosis: too many jobs at once (plan, explore, spawn, monitor, review, edit, merge, decide when to stop). One agent was doing a whole management chain's work.

4. A tree of planners, workers and handoffs. The final design split those jobs back apart, but this time into a hierarchy that can grow. That's the design the rest of this issue is about.

How the Final System Works

Following the numbers in the diagram:

You write the instructions. A human gives the system its goal and constraints. Once a run starts, Lin told Simon Willison, "there's actually no way to steer it." The only control is stopping it, so the instructions carry a lot of weight (more on that below).
The root planner owns the whole scope. It studies the current state of the repo and writes specific, targeted tasks that move toward the goal. It writes no code itself, and it doesn't know or care which agent picks up each task.
Planners spawn subplanners. When a planner decides its scope can be split, it creates subplanners that each fully own one slice, say the CSS engine. This is recursive, so planning itself runs in parallel, and no single planner gets tunnel vision on a huge project.
Workers take one task each. A worker drives its task to completion and knows nothing about the larger system. It doesn't talk to other workers or planners, and it works on its own copy of the repo.
Workers push to one shared branch. There's no central merge gate. Workers resolve their own conflicts.
A handoff goes back up. When a worker finishes, it writes a single handoff that the system delivers to the planner that asked for the task. The handoff covers what was done plus notes, concerns, deviations, findings and feedback. The planner gets it as a follow-up message, pulls the latest code and plans again.

Step 6 is the subtle part. Even a planner that thinks it's "done" keeps receiving handoffs, so it keeps planning. Information flows up the tree toward agents with an increasingly global view, without any global synchronization or cross-talk between workers. Lin notes that this looks a lot like how some human software teams work, even though the models were never trained to organize this way.

Why there's no integrator. Cursor originally added an integrator agent for quality control and to serialize the pushes, rebases and merges from hundreds of workers. It became the obvious bottleneck: hundreds of workers and one gate. Prompt changes didn't fix it, so the team removed it. Workers turned out to be capable of handling conflicts themselves. And conflicts were rarer than you might expect. Lin told Willison that most commits had no merge conflicts, because planners split the work so that tasks rarely overlapped in time or files.

Different models for different roles. Cursor found OpenAI's GPT-5.2 better at extended autonomous work than Claude Opus 4.5, which "tends to stop earlier and take shortcuts." GPT-5.2 was also a better planner than the coding-specialized GPT-5.1-Codex, because the job was "more expansive than merely coding," Lin said: operating inside a harness without a user to ask. The system now picks a model per role.

Throughput Over Perfection

The most counterintuitive decision in the project is about correctness. At first, Cursor required 100% correctness before every commit. That caused heavy serialization. One small error, like an API change or a typo, could halt the whole system. Workers wandered outside their scope to fix unrelated breakage, and many agents piled onto the same issue and trampled each other.

So the team allowed some slack. If a commit introduces a small error, agents trust that another agent will fix it soon, which is true because every part of the codebase has an owner in the tree. The result, in Lin's words, is an error rate that "remains small and constant," not one that grows. He suggests the ideal setup accepts some error rate and adds a final "green" branch, where an agent regularly takes a snapshot and does a quick fix-up pass before release.

The same logic applies when two agents touch the same file. Rather than over-engineer a prevention scheme, the harness accepts "moments of turbulence" and lets the code settle. That costs some tokens but keeps the system simple enough to align and observe.

The numbers from the stable run: about 1,000 commits per hour across 10 million tool calls over one week, with no human intervention after launch. By the time Willison wrote about it, the repo had nearly 30,000 commits.

This is a classic distributed-systems trade-off, made consciously: give up per-commit consistency to get throughput, then reconcile at the end. It's similar to eventual consistency in a database, applied to a codebase.

The Machine Became the Bottleneck

Cursor deliberately avoided building a distributed system. Each multi-agent run had its own large machine, which made metrics easy to watch and state easy to share. Lin told Willison each machine ran about 300 agents, which works because agents spend much of their time thinking rather than running tools. To go beyond one machine, the team split the browser into separate work streams (for example, CSS) and ran one harness per stream on its own machine.

Scaling up exposed bottlenecks that a single developer never sees:

RAM first. After agent memory was capped, the hotspot moved.
Then disk. With a monolithic project, hundreds of agents compiling at once read and wrote many GB/s of build artifacts. Compilation, not thinking, came to dominate time. A later run restructured the project into many self-contained crates and spent far less time waiting on builds, running at "multiple times more throughput."
Then shared locks in tools. Git and Cargo use shared locks as simple concurrency control. The team noticed contention from many agents running git restore at once.

Lin's takeaway: codebase structure affects token throughput. He floats borrowing from databases and storage systems: finer concurrency control instead of coarse tool locks, and copy-on-write plus deduplication for hundreds of near-identical repo copies.

Feedback loops kept agents honest. A week-long autonomous run needs signals that tell agents whether they're right. FastRender used three, according to Lin's interview with Willison:

Specs in the repo. Git submodules pulled in the CSS, HTML, DOM and ECMAScript specifications, and the agent-written code often cites them.
Screenshots vs. golden images. GPT-5.2 can read images, so agents compared rendered screenshots against golden samples.
The Rust compiler. "You can get a lot of verification just from compilation."

Instructions Get Amplified

The harness turned out to be the easy part. The instructions were harder. A multi-agent run is like a normal coding agent with "orders of magnitude more time and compute," Lin writes, and that "amplifies everything, including suboptimal and unclear instructions." Agents follow instructions to the end, good or bad.

Examples from the browser project:

"Spec implementation" was vague enough that agents dove into obscure, rarely used features instead of prioritizing.
Performance expectations were assumed, not stated. It took explicit instructions and enforced timeouts to get agents to balance speed with other goals.
With no stated dependency policy, agents pulled in libraries they could have written. One added QuickJS to unblock itself while other agents were still building the JavaScript engine. A later run listed the banned libraries, which fixed it.

From all this, Cursor drew a few prompting rules that apply well beyond browsers:

Only instruct what the model doesn't know, like multi-agent etiquette or how to run your tests. Treat it like "a brilliant new hire" who knows engineering but not your codebase.
Constraints beat instructions. "No TODOs, no partial implementations" works better than "remember to finish implementations."
Avoid checklists for deep work. A list of specific items makes the model focus on the list and implicitly deprioritize everything else.
Use numbers for scope. "Generate many tasks" gets a cautious handful. "Generate 20–100 tasks" signals ambition, and the team saw very different behavior.

Cursor's January post sums it up: "The harness and models matter, but the prompts matter more."

How Other Teams Coordinate Agents

Cursor's tree isn't the only working design. Comparing it with two Anthropic write-ups shows how much the right structure depends on the task.

Anthropic's C compiler: peers with git locks. Nicholas Carlini ran 16 Claude agents in parallel, each in its own Docker container with its own clone of a shared repo. There's no orchestrator. An agent claims a task by writing a file to current_tasks/, and git's synchronization forces a second agent that claims the same task to pick another one. Over nearly 2,000 Claude Code sessions, two billion input tokens and just under $20,000, they produced a 100,000-line Rust C compiler that builds Linux 6.9 on x86, ARM and RISC-V.

Two of Carlini's lessons mirror Cursor's. First, the verifier decides everything: if it isn't "nearly perfect," Claude "will solve the wrong problem." Second, flat parallelism breaks on one giant task. When all 16 agents hit the same Linux-kernel bug, they overwrote each other's fixes. The fix was GCC as a known-good oracle: compile most kernel files with GCC and a random subset with Claude's compiler, so each agent could chase a different failing file. The test suite, not a planner, splits the work.

Anthropic's Research feature: one lead, parallel subagents. In Claude's Research system, a lead agent plans, spawns 3–5 subagents that search in parallel, and synthesizes their findings. An Opus 4 lead with Sonnet 4 subagents beat single-agent Opus 4 by 90.2% on Anthropic's internal research eval, at about 15× the tokens of a chat. Token usage alone explained 80% of performance variance on BrowseComp. The lead waits for each batch of subagents to finish, which Anthropic calls a bottleneck: the same "slowest worker" problem as Cursor's design 2. Anthropic also notes that most coding tasks have fewer truly parallel pieces than research.

Cursor's own follow-ups. The same harness later took on 235 CUDA kernel problems with NVIDIA, with a planner rebalancing work by benchmark results. Over three weeks it reached a 38% geometric-mean speedup over baselines, and the benchmark threw out any "cheated" result faster than a B200 can physically run. After a February research preview of long-running agents (plan first, then have agents check each other's work), September brought Projects: a coordinator that "doesn't write code itself but directs other agents that do," with shared context files synced to every agent and subscriptions to Slack, schedules and PRs. Cursor says users who mainly work through Projects merge six times as many PRs.

The common thread across all three teams: separate the agents that decide what to do from the agents that do it, give every worker a fresh context and an isolated workspace, and invest heavily in automated checks.

The Trade-offs (and the Skeptics)

The approach has real costs, and the critics have a point.

Code quality. Software Improvement Group ran the public FastRender code through its analysis platform. It scored 1.3 out of 5 for maintainability, in the bottom 5% of systems SIG sees, and 2.1 out of 5 for architecture, meaning tightly coupled components. SIG sized the code at roughly 110 person-years of Rust and warned of "a new class of legacy: large, opaque, and hard to trust." Cursor never presented FastRender as production software, and the browser's own quirks (JavaScript turned off behind a feature flag, vendored layout libraries) show it.
"From scratch" is debatable. The agents pulled in dependencies such as Taffy for flexbox and grid layout. Critics quoted by SIG note that the agents' planning documents used Servo and Chromium as guides.
Cost. "Trillions of tokens," Cursor writes. SIG cites estimates of 10 to 20 trillion, several million dollars at list prices. Even Anthropic's far smaller compiler run cost about $20,000. The work has to be worth the tokens.
Verification is the ceiling. Carlini's compiler started breaking existing features with each new one near the end of the project. Cursor's final "green branch" pass is still a proposal. Carlini, who used to work in penetration testing, worries about "programmers deploying software they've never personally verified."
No mid-run steering. The research harness could only be stopped, not redirected. Projects adds a coordinator you can chat with mid-run.

Key Takeaways

Structure beats self-organization. Equal agents sharing a lock file were slow (20 agents ≈ 2–3) and risk-averse. Clear ownership through planners, workers and handoffs is what scaled.
Separate deciding from doing, and keep each role small. One agent juggling planning, merging and judging broke down. Recursive subplanners and narrow workers didn't.
Throughput needs slack. Accepting a small, stable error rate plus a final fix-up pass beat requiring every commit to be green. Removing the central integrator also helped.
The environment is part of the system. Disk I/O, tool locks, build structure and feedback signals (specs, screenshots, compilers, test oracles) shaped results as much as the models did.
Instructions get amplified. At thousands of agents, vague specs become wasted weeks. Use constraints, explicit numbers and a stated policy for dependencies and performance.

Why Can't LLMs Count the R's in "Strawberry"? How Tokenizers Work

AI Engineering Illustrated — Tue, 13 Oct 2026 22:49:40 +0000

The preprocessing step behind spelling slips, shaky arithmetic, glitch tokens and your API bill

Ask a language model how many r's are in "strawberry" and, for a long time, the popular answer was "two." The same models could write a sonnet about strawberries, explain their genetics and translate the word into forty languages. So why does counting letters go wrong?

The model never sees the letters. Before your prompt reaches the first transformer layer, a separate program called the tokenizer chops the text into chunks and swaps each chunk for an integer. The model only ever reads those integers. If "strawberry" becomes a single integer, the three r's are hidden inside a number the model can't take apart.

Tokenization is the least glamorous part of the LLM stack, and it causes a surprising number of problems: shaky multi-digit arithmetic, higher costs for non-English users, strange "glitch" words that make models act up, and token counts that don't match your character-based estimates. In this issue we'll open the tokenizer up: how it turns text into IDs, how Byte Pair Encoding (BPE) learns its vocabulary, and how its choices show up as spelling, arithmetic, language and glitch-token problems you can design around.

Every token split in this issue was measured with OpenAI's open-source tiktoken library on the actual GPT-2, GPT-4 (cl100k_base) and GPT-4o (o200k_base) vocabularies, so you can reproduce them yourself.

Tokens: the Model's Only Alphabet

A neural network can't do math on strings. It needs numbers. The simplest scheme would give every character a number, but then a 2,000-word document becomes more than 10,000 positions, and every transformer layer pays for each one. The other extreme gives every word a number, but no fixed list covers every word, name, typo, URL and code identifier people will ever type.

Rico Sennrich and colleagues measured this trade-off in their 2016 paper Neural Machine Translation of Rare Words with Subword Units. On a German training corpus of about 100 million words, keeping whole words needed 1.75 million vocabulary entries and still left 1,079 unknown words in the test set. Splitting into characters needed only about 3,000 symbols but stretched the corpus to 550 million positions. Their subword method, BPE, landed in between: about 63,000 symbols, 112 million positions and zero unknown words.

That middle ground is a token: a chunk of text that's usually shorter than a word for rare words and exactly one word for common ones. Modern LLMs use vocabularies of 50,000 to 200,000 tokens. tiktoken's README puts the average at about 4 bytes of text per token. On the English text of last week's issue I measured 4.4 characters per token with the GPT-4o tokenizer, or roughly 0.63 words per token.

From Text to Token IDs in Five Steps

From raw text to token IDs: pre-split, apply BPE merges, look up IDs, then embed

Here's what happens to the sentence "I love strawberry" on its way into GPT-4o:

Raw text. Your string arrives as UTF-8 bytes. Some tokenizers normalize it first (Unicode normalization, lowercasing in older models), but GPT-style tokenizers keep the bytes exactly as typed.
Pre-split. A regular expression cuts the text into pieces that merges are never allowed to cross: letter runs, digit runs, punctuation, whitespace. The tiktoken source shows the patterns. GPT-2's version keeps a leading space attached to the following word, so the pieces are I, love, strawberry. That's why the dot in the figure matters: " strawberry" with a space and "strawberry" without one are different tokens.
BPE merges. Each piece starts as a sequence of bytes. The tokenizer then applies its learned merge rules, in the order they were learned, gluing adjacent pairs together until no rule applies. Common words collapse into one token. Rare ones end up as a few subword chunks.
Token IDs. Each final chunk is looked up in the vocabulary. With o200k_base, "I love strawberry" becomes [40, 3047, 101830]: three integers.
Embeddings. Inside the model, each ID selects one row of a learned embedding matrix, a vector of a few thousand numbers. From here on, the transformer only ever works with those vectors.

Two properties fall out of this design, and the tiktoken README calls both out. Tokenization is lossless: decoding the IDs gives back the exact original bytes. And it works on any text, even strings that never appeared when the tokenizer was trained. The second property depends on a trick we'll get to in a moment.

How BPE Learns Its Vocabulary

Byte Pair Encoding started life as a data compression algorithm (Philip Gage, 1994): repeatedly find the most common pair of adjacent bytes and replace it with a new, unused symbol. Sennrich et al. adapted the idea to build a vocabulary instead of compressing a file. Training goes like this:

Split every word in the training corpus into characters. The starting vocabulary is just those characters.
Count every pair of adjacent symbols, weighted by how often each word appears.
Merge the most frequent pair into a new symbol, add it to the vocabulary, and record the rule.
Repeat until the vocabulary reaches the target size.

The number of merges is the only real setting. As the paper puts it, the final vocabulary size equals the starting symbols plus the number of merge operations.

BPE training: count adjacent pairs, merge the most frequent, repeat until the vocabulary is full

The diagram uses the toy corpus from the Hugging Face LLM course: "hug" (10 times), "pug" (5), "pun" (12), "bun" (4) and "hugs" (5).

Start: the vocabulary is b g h n p s u.
Merge 1: the pair u + g appears 20 times (10 in hug, 5 in pug, 5 in hugs), more than any other, so ug becomes a token.
Merge 2: u + n now leads with 16 (12 in pun, 4 in bun), so un joins.
Merge 3: h + ug appears 15 times, and the first three-letter token, hug, is born.

Real tokenizers run this loop tens of thousands of times over gigabytes of text. GPT-2's vocabulary has 50,257 entries. GPT-4's cl100k_base has about 100,000 and GPT-4o's o200k_base about 200,000 (100,277 and 200,019 including special tokens, according to tiktoken). Meta says Llama 3 moved to a 128K-token vocabulary that yields up to 15% fewer tokens than Llama 2 on the same text.

Encoding new text replays the merges. Split the input into characters (or bytes), then apply the learned rules in priority order. A word seen often during training collapses into one token. A word never seen still gets encoded, just in smaller pieces. With GPT-4o's vocabulary, "unbelievably" becomes un | bel | ievably.

The order of merges also explains some quirks. Merges reflect the tokenizer's training data, which isn't necessarily the model's training data. Keep that in mind; it comes back when we get to glitch tokens.

Byte-Level BPE: No More "Unknown Word"

Classic BPE starts from characters, and Unicode defines roughly 150,000 of them. Any character missing from the training corpus becomes [UNK], the "unknown" token. The Hugging Face course notes this is one reason older NLP models handled emoji badly.

GPT-2 introduced a simple fix: run BPE over bytes instead of characters. There are only 256 possible byte values, so the base vocabulary has 256 entries and every possible input, whether emoji, Tamil script or binary garbage, can be written in them. Merges then build up common multi-byte sequences. As the course puts it, GPT-2 doesn't actually have an unknown token, because an unknown character can't occur with byte-level BPE.

You can see the byte fallback at work. In GPT-2's vocabulary, the emoji 🙂 (four UTF-8 bytes) becomes 2 tokens. GPT-4o's larger vocabulary learned the whole sequence, so it's 1 token. Either way, nothing is ever "unknown." The price is that a rarely seen script can get chopped into fragments that aren't even whole characters, which leads to the language tax below.

Why the Model Can't See the R's

Back to the strawberry. Here's what GPT-4o's tokenizer does with the classic question:

The model sees “ strawberry” as one ID with no letters inside; spelling it out or calling code exposes the letters

"How many r's are in strawberry?" becomes 8 tokens: How | ·many | ·r | 's | ·are | ·in | ·strawberry | ?. The word you care about is token #101830. To the model, that's one opaque symbol, mapped to one embedding vector. There's no "r" in it to count.

Whatever the model knows about how strawberry is spelled, it had to memorize from training text where the word happened to be spelled out, as in dictionaries, spelling-bee pages or kids' worksheets. That knowledge is thin and indirect, so the model guesses, and a plausible guess for a word ending in a double letter is "two."

The split also depends on context in ways people don't expect:

strawberry (leading space, the usual case mid-sentence) is 1 token.
strawberry (no leading space, for example at the start of a message) is 3 tokens: st | raw | berry.
Strawberry (capitalized) is a different 3: Str | aw | berry.

So the model may have learned slightly different things about the same word.

The fixes all expose the letters. Writing the word as s-t-r-a-w-b-e-r-r-y turns it into 10 tokens, one per letter, and now counting is something the model can actually do. Better still, when exact character-level answers matter (counting, reversing a string, checking a regex), have the model call a code tool. One line of Python, "strawberry".count("r"), returns 3 every time.

The Multilingual Token Tax

The tokenizer's merges come from its training data, and that data is mostly English and code. English words get dedicated tokens. Other languages, especially ones with their own scripts, get fewer merges and are left closer to raw bytes.

The token tax: the same sentence costs more tokens in some languages, and numbers split into left-aligned chunks

I ran one short sentence, "The weather is nice today," and simple translations of it through three of OpenAI's tokenizers. In English, it's 6 tokens in all three. In Tamil, it's 85 tokens with GPT-2's tokenizer, 42 with GPT-4's and 11 with GPT-4o's. Hindi drops from 27 to 21 to 5, and Russian from 27 to 13 to 6. That's just one sentence, not a benchmark, but the direction is clear. Larger, more multilingual vocabularies have closed much of the gap, though not all of it.

Aleksandar Petrov and colleagues studied this systematically in Language Model Tokenizers Introduce Unfairness Between Languages (2023). They found the same text can differ in token length by up to 15× across languages, and that even byte-level and character-level models show gaps of more than 4×. The consequences are concrete:

Cost: APIs bill per token.
Context: a 128K-token window holds far fewer pages of Tamil than of English.
Latency: output is generated one token at a time.

If you serve a non-English market, measure token counts on your actual traffic. Don't assume the "about 4 characters per token" rule of thumb.

Numbers: Why Long Addition Goes Wrong

The same pre-split regex decides how numbers are chunked, and labs have made very different choices. Aaditya Singh and DJ Strouse surveyed them in Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs (2024):

Pure BPE (GPT-3, Claude 2.1, OLMo): whichever digit strings happened to be frequent got tokens. 710 might be one token while 711 isn't, so numbers of the same length split inconsistently.
Single digits (PaLM, Llama 1 and 2, Mistral): every digit is its own token.
Chunks of up to 3 digits, left to right (GPT-3.5, GPT-4): the \p{N}{1,3} rule in the tiktoken pattern. GPT-4o's o200k_base uses the same rule.

Left-to-right chunking means 1234567 becomes 123 | 456 | 7. The chunks line up with the start of the number, while place value, and the way carries move when you add, runs from the end. Insert commas and the tokenizer splits on them, giving 1 | , | 234 | , | 567. Now every chunk sits on a thousands boundary.

Singh and Strouse tested exactly this on GPT-3.5 with 7- to 9-digit addition. With plain digits, 8-shot accuracy was 75.6%. With commas, it was 97.8%. When the answer had more digits than the inputs (because of a final carry), plain-digit accuracy fell to 8.25%, and the errors weren't random: the model almost always got the first three digits right and the fourth digit wrong, right at the first token boundary. Asking the model to first rewrite the problem with commas recovered most of the lost accuracy. GPT-4 was less affected, but the effect didn't disappear.

The lesson isn't "add commas everywhere." It's that number formatting changes the problem the model has to solve. For anything that must be exact, like invoices or financial math, route the arithmetic to a calculator or code tool.

Glitch Tokens: When the Vocabulary Outlives the Data

In early 2023, Jessica Rumbelow and Matthew Watkins went looking at GPT-2 and GPT-3's token embeddings and found a cluster of strange entries: SolidGoldMagikarp, TheNitromeFan, cloneembedreportprint, petertodd. In their LessWrong write-up, they showed that asking a model to simply repeat these strings produced evasion, insults or substitutions. ChatGPT, for example, repeated " SolidGoldMagikarp" as "distribute." Some of these tokens even made GPT-3 give different answers at temperature 0, which is supposed to be deterministic.

Their partial explanation goes back to the point about merges. The tokenizer was trained on scraped web text that included Reddit usernames, e-commerce site internals and game logs. The model was trained on more heavily curated text. So strings like SolidGoldMagikarp (I confirmed it's a single token, #43453, in GPT-2's vocabulary) earned a vocabulary slot but almost never showed up during model training. Their embedding vectors stayed close to wherever they started, and the model doesn't know what to do when it meets one.

It's not just a GPT-2 curiosity. Fishing for Magikarp (Land and Bartolo, 2024) built automatic detectors that combine tokenizer analysis, weight-based signals and prompting, and found under-trained tokens across a wide range of models. In newer OpenAI vocabularies, " SolidGoldMagikarp" is 5 ordinary tokens (·Solid | Gold | Mag | ik | arp), but each new tokenizer can bring its own oddities. If you train or fine-tune models, scanning for under-trained tokens is cheap insurance.

The Other Tokenizer Families

BPE dominates today's LLMs, but you'll run into two other algorithms, often through one library:

WordPiece (developed by Google for BERT) also learns merges bottom-up, but according to the Hugging Face course it scores a pair as its frequency divided by the product of its parts' frequencies, which favors pairs whose parts are rare on their own. It keeps only the final vocabulary and encodes each word by taking the longest piece it knows, then the next. Pieces inside a word carry a ## prefix, as in token ##ization. (Google never open-sourced its training code, so the course describes a best-guess reconstruction.)
Unigram LM, from Taku Kudo's Subword Regularization (2018), works top-down. It starts with a large candidate vocabulary, scores how much each piece contributes to a probabilistic model of the corpus, and prunes the least useful pieces. Since the same word can be segmented several ways, Kudo trained translation models on sampled segmentations as a kind of noise, which made them more robust, especially in low-resource and out-of-domain settings.
SentencePiece (Kudo and Richardson, 2018) is a library rather than an algorithm. It implements both BPE and Unigram and trains directly on raw sentences, with no language-specific pre-splitting. Whitespace becomes the visible symbol ▁, so I saw a girl becomes ▁I ▁saw ▁a ▁girl (example from the SentencePiece README). That makes it a natural fit for languages that don't put spaces between words, like Japanese and Chinese.

Picking a Vocabulary Size

Why not a million tokens? Every entry costs a row in the input embedding matrix and another in the output layer that scores the next token (unless the two are tied). With 4,096-dimensional vectors, a 200,000-entry vocabulary is about 820 million parameters per matrix. More entries also means fewer training examples per rare token, which is the glitch-token problem again.

Smaller vocabularies mean longer sequences: attention costs more, the context window fills faster and generation takes more steps. Recent models have moved toward larger vocabularies (50K, then 100K, 128K and 200K) as the embedding table becomes a small share of total parameters. Meta credits Llama 3's 128K tokenizer with "substantially improved model performance."

A Cheat Sheet for Engineers

Common tokenizer gotchas, their causes and practical fixes

A few of these deserve a sentence more.

Count tokens with the real tokenizer. Use tiktoken.encoding_for_model("gpt-4o") for OpenAI models, or your provider's token-counting endpoint, before you enforce context limits or estimate costs. Character-based estimates are wrong for code, non-English text and JSON.

Watch the boundary at the end of your prompt. Because GPT-style tokenizers attach the space to the following word, " Paris" is a single token. A prompt that ends with a trailing space ("The capital of France is ") leaves a bare space token, and the model then has to continue with "Paris" without its usual leading space, an unusual sequence it saw less often in training. Trim trailing whitespace from completion-style prompts. The "token healing" technique cited by Singh and Strouse goes further: it backs up a few tokens and lets the model regenerate across the boundary.

Special tokens are control characters. Strings like <|endoftext|> map to reserved IDs (100257 in cl100k_base, 199999 in o200k_base) that mark document and message boundaries. By default tiktoken refuses to encode them from ordinary text: encode("hi <|endoftext|>") raises an error. Keep that protection on, so pasted text can't forge a document or message boundary.

Key Takeaways

LLMs read integers, not letters. A tokenizer turns text into IDs before the model runs, and anything inside a token, like its spelling, is invisible unless the model memorized it.
BPE builds the vocabulary by merging the most frequent adjacent pairs, and byte-level BPE guarantees that any input can be encoded with no "unknown" token.
Tokenization explains classic failure modes: letter counting, multi-digit arithmetic (75.6% vs 97.8% for GPT-3.5 depending only on commas), higher costs for some languages (up to 15× longer encodings) and glitch tokens.
The fix is usually to change the representation or hand off the work: spell words out, format numbers deliberately, and call code for anything that must be exact.
Measure with the actual tokenizer. "4 characters per token" is a rough English-only average, not a budget.

How LLM Inference Serving Works: KV Cache, PagedAttention and Continuous Batching

AI Engineering Illustrated — Mon, 12 Oct 2026 13:21:24 +0000

Why serving a model is mostly a memory-management problem, and the operating-systems idea that fixed it

Training a large language model gets the headlines, but inference is where most of the compute bill goes in production. Every chat message, every agent step and every autocomplete is a request for a GPU to generate tokens. The difference between a naive serving stack and a good one isn't 10%. The vLLM team reported up to 24× higher throughput than serving with Hugging Face Transformers when they launched in June 2023.

Where does a gap that large come from? Not from faster matrix multiplication. It comes from three things most application developers never see:

How the KV cache is stored in GPU memory
How requests are batched together
How work is scheduled between the two phases of generation

In this issue, we'll look at:

Why generation has two very different phases, prefill and decode
What the KV cache is, and how large it really gets (with exact numbers for Llama 3.1 8B)
How PagedAttention borrowed virtual memory from operating systems to stop wasting most of that memory
Why continuous batching replaced static batching
How prefix caching and chunked prefill squeeze out more performance, and what that means for how you write prompts

Two Phases: Prefill and Decode

An LLM generates text one token at a time. Each new token depends on all the previous ones. Under the hood, a single request goes through two phases with very different performance profiles.

Prefill processes the entire prompt in one forward pass. All prompt tokens go through the model in parallel, which means large matrix multiplications that keep the GPU's compute units busy. Prefill is compute-bound. Its output is the first generated token, so prefill time largely sets time-to-first-token (TTFT).

Decode generates every token after that, one forward pass per token. Each step processes only one new token per request, but to compute attention it has to read the keys and values of every previous token. So each step loads the model's weights plus the request's entire cached history from GPU memory, does relatively little math with it, and produces one token. Decode is memory-bound: the bottleneck is how fast bytes move from GPU memory (HBM), not how fast the GPU computes. vLLM's own tuning docs describe the split exactly this way: "compute-bound (prefill) and memory-bound (decode)".

This asymmetry drives almost every design decision that follows. Since decode is limited by memory bandwidth, the main way to get more out of a GPU is to decode many requests at once. Loading the weights once and using them for 64 requests is far better than loading them 64 times. Batch size is the lever, and what limits batch size is memory, specifically the KV cache.

What the KV Cache Is, and How Big It Gets

In a transformer's attention layers, each token is projected into a query, a key and a value. When generating token n, the model compares token n's query against the keys of tokens 1…n−1 and takes a weighted mix of their values. Those keys and values never change once computed, so recomputing them at every step would be wasteful. Instead, inference engines store them in the KV cache. Prefill fills the cache for the prompt, and each decode step appends one more token's K and V.

How much memory is that? It's a simple product:

KV bytes per token = 2 (K and V) × layers × KV heads × head dimension × bytes per value

Plugging in the published config.json for Llama 3.1 8B: 32 layers, 8 key/value heads (the model uses grouped-query attention, so its 32 query heads share 8 KV heads), head dimension 128, and bf16 values at 2 bytes each:

2 × 32 × 8 × 128 × 2 = 131,072 bytes = 128 KiB per token

That sounds small until you multiply by context length:

1K tokens → 128 MiB
8K tokens → 1 GiB
32K tokens → 4 GiB
128K tokens (the model's maximum context) → 16 GiB

The model's weights are about 8 billion parameters × 2 bytes ≈ 16 GB. A single conversation at full context needs about as much memory as the entire model. The weights are shared by every request. The KV cache is per request, and it grows with every token generated.

(Grouped-query attention is itself a KV-cache optimization. With 32 KV heads instead of 8, every number above would be four times larger.)

The vLLM launch post gives a sense of scale for an older model: the KV cache can take up to 1.7 GB for a single sequence in LLaMA-13B. It also describes the cache as dynamic: "its size depends on the sequence length, which is highly variable and unpredictable." That unpredictability is the real problem.

The Problem: Fragmentation

When you start generating, you don't know whether the answer will be 20 tokens or 2,000. Early serving systems handled this the obvious way. They reserved a contiguous chunk of GPU memory for each request, big enough for the maximum possible length.

The top half of the diagram shows what goes wrong:

Over-reservation. A request that ends after 200 tokens still held space for 2,048. That reserved but unused memory can't go to anyone else.
Internal fragmentation. Even within a reservation, the unused tail sits idle the whole time the request is running.
External fragmentation. Contiguous allocations of different sizes leave gaps between them that are too small to use.

The vLLM team measured this: existing systems wasted 60%–80% of KV cache memory due to fragmentation and over-reservation. Since memory caps batch size, and batch size caps decode throughput, most of the GPU's potential was being thrown away.

PagedAttention: Virtual Memory for the KV Cache

Operating systems solved this exact problem decades ago. Programs want large contiguous address spaces, but physical memory is fragmented. The OS splits memory into fixed-size pages, gives each process a contiguous virtual address space, and keeps a page table mapping virtual pages to wherever they physically sit.

PagedAttention, introduced in the vLLM paper (SOSP 2023, arXiv:2309.06180), applies the same idea to the KV cache. In the authors' words: "one can think of blocks as pages, tokens as bytes, and sequences as processes."

The bottom half of the diagram shows how it works:

KV memory is split into fixed-size blocks, each holding the keys and values for a fixed number of tokens (configurable in vLLM with --block-size).
Each request sees a sequence of logical blocks that are contiguous from its own point of view.
A block table maps each logical block to a physical block anywhere in GPU memory. Physical blocks don't need to be adjacent.
Physical blocks are allocated on demand, only when a request's current block fills up.
The attention kernel is rewritten to follow the block table, gathering each request's K and V from wherever its blocks live.

The payoff: memory waste only happens in a request's last, partially filled block. The vLLM post reports this results in "a mere waste of under 4%". Nearly all KV memory now holds real data, so far more requests fit in a batch. The paper's headline result is that vLLM "improves the throughput of popular LLMs by 2-4× with the same level of latency" compared with FasterTransformer and Orca, with bigger gains for longer sequences, larger models and more complex decoding.

Sharing comes for free

Once memory is indirected through a block table, two requests can point at the same physical block. This matters for decoding strategies that create several outputs from one prompt, like parallel sampling ("give me 4 candidate answers") or beam search. The prompt's KV blocks are stored once and shared, with reference counts. A block is copied only when one branch needs to write to it (copy-on-write, another OS technique). The vLLM team reports this cuts memory usage of these methods "by up to 55%", which can mean "up to 2.2× improvement in throughput."

Continuous Batching

Freeing memory only helps if the scheduler actually fills it with work. The classic approach, static or request-level batching, doesn't.

With static batching, the server collects a batch of requests and runs them together until all of them are done. But generation lengths vary wildly. The Orca paper (OSDI '22) describes the problem precisely: with request-level scheduling, "requests that have finished earlier than other requests in a batch cannot return to the client, while newly arrived requests have to wait until the current batch completely finishes."

Orca's fix was iteration-level scheduling, now usually called continuous batching. The scheduler re-forms the batch at every model iteration, meaning every decode step:

A request that just produced its final token leaves the batch immediately and its result is returned.
A newly arrived request joins at the very next iteration.
The batch stays full, so the GPU stays busy.

Continuous batching and PagedAttention reinforce each other. Continuous batching constantly admits and evicts requests of unpredictable length, which is exactly the workload that fragments contiguous memory. Paged blocks make admitting a new request cheap: you just need a few free blocks, not a large contiguous region.

When memory runs out anyway

Even with paging, a burst of long requests can fill the KV cache. vLLM then preempts some requests to free their blocks. In vLLM V1 the default preemption mode is recompute: the preempted request's KV is dropped and rebuilt later, rather than swapped to CPU memory, "as recomputation has lower overhead in the V1 architecture." If your logs show preemption warnings, the docs suggest giving vLLM more memory (gpu_memory_utilization, which defaults to 0.92), reducing max_num_seqs or max_num_batched_tokens, or sharding the model across more GPUs.

Prefix Caching and Chunked Prefill

Two more techniques complete the picture. The first one directly affects how you should write prompts.

Automatic prefix caching

Real traffic is highly repetitive at the start. Every request to an agent typically begins with the same system prompt, the same tool definitions, maybe the same retrieved documents. Running prefill on those identical tokens over and over is pure waste.

With prefix caching, vLLM keeps the KV blocks of previous requests and reuses them. The design docs explain how a match is found: vLLM hashes "each kv-cache block by the tokens in the block and the tokens in the prefix before the block." Because the hash includes everything before the block, a block matches only if the entire prefix up to it is identical. A lookup table maps hashes to cached physical blocks, so a new request whose first 3,000 tokens match an earlier one skips straight to prefilling only the new suffix.

Cached blocks no longer in use aren't freed immediately. They stay available for reuse and are evicted in LRU order when memory is needed. For multi-tenant deployments, vLLM supports an optional per-request cache_salt that is mixed into the first block's hash, so different tenants can't reuse, or probe, each other's cached prefixes.

The docs note that prefix caching is "widely used by many public endpoints (e.g., OpenAI, Anthropic, etc.)". This is the mechanism behind the discounted "cached input" pricing those APIs offer.

The practical lesson for prompt design: put stable content first and variable content last. System prompt, then tool definitions, then long reference documents, then the user's actual question. Changing a single token early in the prompt (a timestamp in the system prompt, say) breaks the hash chain from that point on and forces a full prefill.

Chunked prefill

Remember that prefill is compute-heavy and decode is memory-bound. If a 10,000-token prompt arrives while 60 users are streaming responses, a naive scheduler runs that huge prefill as one step, and every streaming user sees a stall.

Chunked prefill splits large prefills into pieces and mixes them into steps alongside decodes. vLLM's docs describe the V1 policy:

Each step has a token budget, max_num_batched_tokens.
The scheduler prioritizes decode requests: it "batches all pending decode requests before scheduling any prefill operations."
Any remaining budget goes to pending prefills. A prefill that doesn't fit "automatically" gets chunked.

This improves inter-token latency, because streams don't stall, and GPU utilization, because compute-bound and memory-bound work share each step. The budget is a tuning knob. Smaller values (the docs give 2048 as an example) favor smooth streaming. Larger values favor time-to-first-token, and the docs recommend going above 8192 for throughput on large GPUs. In vLLM V1, chunked prefill "is enabled by default whenever possible."

Putting It Together: The Life of a Request in vLLM

Here's what happens to one request in a modern serving engine:

The request arrives and is tokenized.
The scheduler checks the prefix cache. Matching leading blocks are reused, and only the remainder needs prefill.
At the next iteration, the scheduler admits the request if enough free KV blocks exist, preempting another request if necessary.
Its prefill runs, chunked if it's long, alongside other requests' decode steps, within the token budget.
Each decode step appends one token's K and V into the request's current block. A new physical block is allocated only when that block fills.
When the request emits its stop token, it leaves the batch immediately. Its blocks' reference counts drop, and they become cached-but-evictable.

Each step targets the same scarce resource: GPU memory for KV cache, and keeping the GPU busy with as many requests as that memory allows.

What This Means If You Don't Run GPUs

Even if you only call hosted APIs, these mechanics show up on your bill and in your latency charts:

Long prompts cost twice: compute at prefill (TTFT), and memory for every decode step that follows. Trim context you don't need.
Prompt structure affects cost. Stable prefixes hit the cache. Dynamic content at the top of the prompt defeats it.
Output length affects throughput more than you'd think. Every generated token is a full decode step holding KV memory.
TTFT and tokens-per-second are different metrics driven by different phases. A provider can be strong on one and weak on the other.

Key Takeaways

Inference has two phases: prefill (parallel, compute-bound, sets TTFT) and decode (one token per step, memory-bound).
The KV cache is the bottleneck. For Llama 3.1 8B it's 128 KiB per token, so 16 GiB at a full 128K context, about the size of the model itself.
Contiguous allocation wasted 60–80% of KV memory. PagedAttention uses fixed-size blocks and a block table, like OS virtual memory, cutting waste to under 4% and delivering 2–4× throughput in the paper.
Continuous batching (Orca's iteration-level scheduling) re-forms the batch every step, so finished requests leave and new ones join immediately.
Prefix caching reuses KV blocks for identical prefixes via chained block hashes, so put stable prompt content first.
Chunked prefill schedules decodes first and splits long prompts, so one big prompt doesn't stall everyone else.

DEV Community: AI Engineering Illustrated

How Cursor Got Thousands of AI Agents to Build One Browser

Background: Why One Agent Isn't Enough

Four Harness Designs, Three Failures

How the Final System Works

Throughput Over Perfection

The Machine Became the Bottleneck

Instructions Get Amplified

How Other Teams Coordinate Agents

The Trade-offs (and the Skeptics)

Key Takeaways

Further Reading (sources used)

Why Can't LLMs Count the R's in "Strawberry"? How Tokenizers Work

Tokens: the Model's Only Alphabet

From Text to Token IDs in Five Steps

How BPE Learns Its Vocabulary

Byte-Level BPE: No More "Unknown Word"

Why the Model Can't See the R's

The Multilingual Token Tax

Numbers: Why Long Addition Goes Wrong

Glitch Tokens: When the Vocabulary Outlives the Data

The Other Tokenizer Families

Picking a Vocabulary Size

A Cheat Sheet for Engineers

Key Takeaways

Further Reading (sources used)

How LLM Inference Serving Works: KV Cache, PagedAttention and Continuous Batching

Two Phases: Prefill and Decode

What the KV Cache Is, and How Big It Gets

The Problem: Fragmentation

PagedAttention: Virtual Memory for the KV Cache

Sharing comes for free

Continuous Batching

When memory runs out anyway

Prefix Caching and Chunked Prefill

Automatic prefix caching

Chunked prefill

Putting It Together: The Life of a Request in vLLM

What This Means If You Don't Run GPUs

Key Takeaways

Further Reading (sources used)