DEV Community: Bato

From GPT-2 to Kimi K3: How Language Models Learned to Manage Memory

Bato — Sat, 15 Aug 2026 00:00:16 +0000

Six architectures, one idea: from the KV cache to Kimi K3's hybrid of recurrent memory, global attention, and sparse experts, each a smarter way to manage a limited memory.

TL;DR: In 2019, GPT-2 (small) had 124 million parameters. In 2026, Kimi K3 has 2.8 trillion, about 22,580× more. Yet the leap was never only about size. A large part of the story is quieter: how models learned to remember, update, and retrieve information more efficiently. This post traces that one thread across six architectures, starting with the model that makes the idea easiest to see, GPT-2.

This piece is inspired by Ali Taha's excellent deep-dive "22,580: GPT-2 to Kimi K3, explained" on the Baseten blog (the six-stage structure and the 22,580 comparison follow it closely), and by the Kimi K3 paper, "Kimi K3: Open Frontier Intelligence".

1. GPT-2: the standard Transformer

GPT-2 is a decoder-only Transformer, the architecture from 2017's Attention Is All You Need. "Decoder-only" means it does exactly one job: predict the next token from the tokens before it. There's no separate "encoder" stage reading a source sentence the way a translation model would; GPT-2 simply continues a sequence. When the model processes any position, it is only allowed to look at that position and the ones before it, never the future.

GPT-2 works autoregressively: it predicts one token, appends it to the sequence, and then uses the whole updated sequence to predict the next one, and repeats. Every prediction rests on everything that came before it: the original prompt plus each token generated so far.

Walk through the prompt "The cat sat on the". To choose the next word, the model processes those five tokens and, from the representation at the last one (the), predicts a likely continuation, say mat. That token is appended, the model runs again (now able to attend to all six tokens), and so on. Two things are worth noticing. First, GPT-2 doesn't squeeze the past into a single running summary the way older recurrent networks (RNNs) did; it keeps a separate, directly addressable representation for every token in the window. Second, it handles the prompt in one parallel pass ("prefill"), then generates one token at a time ("decoding").

The mechanism that lets a token "read" the others is called attention. At each position, the model scores how relevant each earlier token is (and the current position itself), then blends their information in proportion to those scores, so relevant tokens contribute more and irrelevant ones less. It's the representation at the current position that carries this blended look-back forward to predict the next token. This weighted look-back is the single most important operation in a Transformer, and it is the one the later architectures work hardest to make cheaper.

Figure 1. Predicting the word after "The cat sat on the." Attention runs from the **last input token* (the), which may look at itself and all earlier tokens; the representation it produces is what predicts the next token, mat. Note that mat has no representation of its own until it is generated and fed back in.*

The hidden inefficiency

Attention at a given position needs a small bundle of information from itself and every earlier token. (Exactly what that bundle contains comes up in a moment.) If the model rebuilt that bundle for every past token from scratch at each step, it would redo almost the same work again and again, and the redundant work would pile up faster the longer the text became.

Figure 2. Each colored cell is one token's key/value being computed in a given step. Only one is genuinely new per step (green); the rest are **recomputations* of key/value projections already done (red). Rebuilding them from scratch costs about 1 + 2 + 3 + 4 = 10 projection computations just to emit 4 tokens, and grows like n². (This counts key/value projection work, not the attention comparison itself, which comes up below.)*

The fix: the KV cache

The fix starts with naming that "bundle of information." Inside every attention layer, each token's hidden representation is passed through three small learned transformations that produce three vectors:

Q, the query: "what am I looking for?"
K, the key: "how should I be found by others?"
V, the value: "what information do I pass on if I'm selected?"

Attention compares the current token's Q against the K of every earlier token (and its own) to decide how much to draw from each, then mixes their Vs accordingly.

K and V are called intermediate representations because they aren't the input (the words) and they aren't the final output (the prediction). They're vectors computed inside the attention layer, on the way from one to the other. And this is what the whole fix hinges on: a past token's K and V never change when a new token arrives. "The cat sat" produces the same K and V for cat whether the next word turns out to be on, quietly, or down. Why? Attention is causal: a token's internal representation is built only from itself and the tokens before it (the mask blocks any peeking ahead), and K and V are computed straight from that representation. So a token that arrives later has no path to reach back and change them.

Because those vectors are stable, there's no need to recompute them. Each token's K and V are computed once and stored; that store is the KV cache. Every new step then computes a fresh Q, K, and V for just the current token, appends its K and V to the cache, and reuses everything already saved, which eliminates the red recomputation triangle in Figure 2.

While the KV cache saves the model from rebuilding past keys and values (the waste in Figure 2), but it doesn't save it from reading them. To produce each new token, the model still has to scan the entire cache: compare the new token against every key stored so far and mix in every value.

And that cache only grows. Writing the 10th token means scanning about 10 entries; the 1,000th token scans about 1,000; the millionth scans about a million. So every new token costs a little more than the last, and generation keeps slowing as the context gets longer. Add all that scanning up over a long passage and the total work grows with the square of the length: double the text and you do roughly four times the attention work (this is attention's well-known quadratic cost). Caching removes the redundant rebuilding; attention's built-in "look back over everything so far" cost stays, and it's a big reason long context is expensive.

Figure 3. Left: every token is projected into Q, K, V inside the attention layer. Right: the K and V vectors are cached and reused, so the cache gains one entry per token.

The catch that shapes everything after GPT-2

The KV cache trades compute for memory, and that trade has a cost: the cache grows with the length of the text. Every new token permanently adds its K and V, so the memory it occupies is roughly:

cache size ≈ (number of tokens) × (size of one token's K,V) × (number of layers).

For a short chat, that's negligible. But that cache typically has to sit in (or stream through) fast accelerator (GPU) memory while the model generates, and that memory is finite and expensive. Now imagine scaling GPT-2's full-attention design to a book-length document, or a million-token codebase: the KV cache alone can swell to many gigabytes, and cache capacity and memory bandwidth can become as much of a bottleneck as the attention math. (GPT-2's own context window was just 1,024 tokens, so this is a thought experiment about its design) Modern long-context models attack this head-on, and Kimi K3 mostly replaces the growing KV cache with a fixed-size recurrent state in the bulk of its layers, keeping compressed global attention in a periodic minority of them.

That tension, direct access to every past token but a cache that grows linearly with context length, is the thread the next architectures pull on. The first attempt, linear attention, gives up some of that direct addressability in exchange for a memory that doesn't grow at all.

2. Linear attention: compress the past into a fixed-size state

Linear attention starts from a blunt question: instead of storing a key and value for every token, what if the model kept a single fixed-size memory and folded each new token into it?

That's the whole idea. Where full attention keeps a KV cache that grows by one entry per token, linear attention maintains one fixed-size state: a memory matrix S that summarizes everything seen so far. Reading and writing never get more expensive, no matter how long the text is.

Figure 4. Full attention stores a key/value entry per token, so memory grows with length. Linear attention folds every token into one fixed-size state, so memory stays constant.

How it works, without heavy math

Recall standard attention: for a new token, you compare its query against every stored key, turn those comparisons into weights, and take the weighted average of the values. That per-query loop over all the keys is the expensive part, and the reason you must keep every key and value around if you want exact attention.

So why can't we just pre-add the past into one running total and reuse it for each new query? It comes down to how the score is computed. In standard attention, the score for a query-key pair is exp(query · key): the query and key are multiplied together and then run through an exponential. That fuses them into a single number, and you can't compute any useful piece of it until you know both the query and the key. So exact softmax attention can't be boiled down to one fixed-size summary that works for every possible future query; each new query has to be re-scored against every stored key.

Linear attention changes the scoring rule to remove that fusion. Instead of the exponential, it applies the same fixed transformation separately to the query and the key, then compares the two transformed vectors with a plain dot product. The point of that transformation is to make the score separable: it splits into a query-only part and a key-only part, instead of one exponential that welds them together.

That separation is the entire trick. Because the query-only part is the same for every past token in a given read, the model doesn't have to pair it with each key one at a time. It can fold all the transformed keys and values into a single running summary as they stream in, and then apply the transformed query to that summary once. In other words: each token adds its transformed key and value to the summary a single time, and a later query simply reads that summary, while a second small running total keeps the result normalized as a weighted average.

Two things fall out of this:

Constant memory (per head and layer, with respect to sequence length). S and z have fixed sizes, set by the model's dimensions, not by how many tokens you've seen. A million-token document uses the same-sized summary as a ten-token one.
It's a recurrence. Each step updates the summary from only the new token and the previous summary, exactly how a recurrent neural network (RNN) carries a hidden state, the very thing Transformers moved away from in Block 1. Linear attention quietly brings that fixed-size memory back. (This equivalence is the point of the aptly named paper Transformers are RNNs.) One nuance: during training or prompt prefill the whole sequence is known, so the summary can still be built in parallel, and linear attention doesn't give up the Transformer's parallel training. It's step-by-step generation that runs as a recurrence.

That fixed state also brings back an RNN-like capacity limit: however long the sequence gets, all of its history has to share the same finite memory. That sharing is where interference creeps in.

For the curious: the exact formulas. Write φ(x) for that fixed transformation (often called a feature map), and let each sum run over the tokens seen up to the current position. Linear attention is just a weighted average that uses the separable score φ(q) · φ(k), and regrouping the sum is what turns it into a reusable summary:

direct form:
  output = Σᵢ [φ(q) · φ(kᵢ)] vᵢ  /  Σᵢ [φ(q) · φ(kᵢ)]

same calculation, regrouped:
  S = Σᵢ φ(kᵢ) vᵢᵀ
  z = Σᵢ φ(kᵢ)
  output = [φ(q) · S]  /  [φ(q) · z]

The point of regrouping: S and z don't depend on the query, so they're built once and reused. This is a different similarity rule, and it doesn't reproduce softmax exactly. The paper's feature map is φ(x) = ELU(x) + 1, whose positive outputs keep every weight ≥ 0 and (after dividing by φ(q) · z) summing to 1, the same two properties softmax gave us.

Figure 5. Linear attention carries a fixed-size state (S, z) from step to step. Each token updates S (green), and each output is read from the current S (blue), the same update-a-hidden-state pattern as an RNN.

The catch: one small table, many memories

Constant memory isn't free. Full attention keeps a separate row for each token, so a query can pick out exactly the ones it needs. Linear attention adds every key-value pair into the same fixed-size table S. If two keys write into overlapping parts of that table, their values mix, so a later query can pull back a blend instead of one clean value. That's interference. And because the basic update only ever adds, with no way to overwrite or forget, the crowding only grows as more pairs share the same fixed space. (This is exactly the capacity problem analyzed in Linear Transformers Are Secretly Fast Weight Programmers, which motivates the next step.)

So the trade from Block 1 flips: full attention gave direct addressability at ever-growing cost; linear attention gives constant cost but crowded, interfering memory. The next three steps attack that crowding. The first asks a sharper question: when we fold a new token into S, are we writing it in the right way? That is DeltaNet.

3. DeltaNet: write the correction, not just the sum

Linear attention's flaw was that it only ever adds. Every token writes its key→value association into S on top of whatever is already there, so associations pile up and interfere. DeltaNet fixes the write: before storing a new association, it checks what the memory already holds for that key and writes only the difference.

That difference is the "delta," and the idea is a classic: the delta rule (Widrow-Hoff, 1960), the same error-correction step at the heart of early machine learning. Applied to our fixed-size memory, it turns a blind add into a deliberate read-modify-write.

Figure 6. Write the same key twice. Plain linear attention keeps both associations, so a later read returns a mixture (interference). DeltaNet instead reads what's stored, computes the correction, and writes only that, moving the entry toward the new value.

One useful picture before the mechanics. Think of S as a compressed, fuzzy dictionary: a key is a learned numeric address, a value is the information written at that address, and a query is a lookup request. Unlike a real dictionary, the addresses aren't isolated slots: nearby keys overlap, so a lookup can return a blend. Two kinds of reads happen against it: the layer's actual output is looked up with a query q, while the delta update below does its own read with the key k, to check what's already stored where it's about to write.

The delta rule in three steps

DeltaNet keeps the same fixed-size matrix S, but changes how a new key-value association is written. First, one setup detail: the modern DeltaNet formulation divides each key by its own geometric length, so every key ends up the same unit size: same direction, standardized magnitude. That common size is what lets the write strength β behave as a clean 0-to-1 dial. (This standardizing step is called L2 normalization, dividing a vector by its length, and is unrelated to L2 regularization.) When token t arrives with key k and value v:

Read the memory at this key: v̂ = k · S. Here k is a vector and S is a matrix, so the result is another vector: the value currently stored at that address.
Compare it to the new target: Δ = v − v̂, the gap between the new target value and what this key currently retrieves.
Write only the correction: S ← S + βₜ · (k ⊗ Δ). The symbol ⊗ is an outer product: it combines the key and correction vectors into a small matrix with the same shape as S, so it can be added straight into the memory.

The write strength βₜ (between 0 and 1) isn't a fixed knob; the model learns it from the current token (βₜ = sigmoid(Wβ · xₜ)), so it can barely touch memory, partially update, or strongly replace. After the write, a read at the same key returns (1−βₜ)·v̂ + βₜ·v: it moves toward the new value, landing exactly on it when βₜ = 1.

What it fixes, and what it doesn't

By correcting instead of piling on, DeltaNet is a much better writer: writing a key again moves what it retrieves toward the new value rather than stacking a second association on top.

But DeltaNet still lacks a separate forgetting control. It can correct what the current key retrieves, but it has no independent way to fade the rest of the state when information becomes stale. Old associations can therefore persist and keep competing for the fixed-size memory. Gated DeltaNet adds a learned decay gate that deliberately fades prior state before the next write.

4. Gated DeltaNet: give the model a forget button

Gated DeltaNet supplies the control DeltaNet was missing: a learned forget gate. To see what it fades (and to follow the upgrade in the next section), it helps to picture the memory concretely.

A key is a vector with dₖ components, called key channels. The memory S is a matrix with one row per key channel (dₖ rows in all). In the read k · S, component kᵢ weights row i, and the weighted rows combine into the retrieved value.

Gated DeltaNet's gate is a single retention factor αₜ between 0 and 1, produced fresh for each token and each head. Before the corrective delta write, it multiplies that head's entire state by αₜ (every row by the same number), so a value near 1 keeps almost all of the memory and a smaller one fades more of it. Then it runs the same read→compare→write step from Block 3.

Let S̄ be the state after fading. Gated DeltaNet reads and corrects this faded state:

S̄ = αₜ · S                     # fade the whole state
v̂ = k · S̄                      # read the faded state
S  = S̄ + βₜ · (k ⊗ (v − v̂))    # corrective delta write

The order matters (fade, then read, then correct), so the comparison is made against the faded state. Because αₜ is learned per token, the gate can output a factor near 1 to preserve the state, or a smaller one (for instance on inputs that mark a context shift) to clear space. It gives DeltaNet the dial it was missing: "how much of the past should I keep?"

Figure 7. Gated DeltaNet inserts one step before the delta write: scale the entire memory by a learned forget gate αₜ. αₜ ≈ 1 applies no global decay; a smaller αₜ forgets aggressively.

Gated DeltaNet combines DeltaNet's corrective write with the same kind of input-dependent scalar state decay used in Mamba-2. Yang, Kautz, and Hatamizadeh formalized the combination in Gated Delta Networks: Improving Mamba2 with Delta Rule (ICLR 2025): a memory that can both correct what a key retrieves and fade the rest on purpose.

The catch: one dial for everything

Gated DeltaNet is a real step up: the model can now edit and forget. But its gate is a single scalar per head (a head is one of the parallel attention sub-units): it fades that head's entire state by one number, so every channel, every row of S, decays at the same rate. That is the limit: one scalar can't keep some rows while quickly fading others within the same head.

The fix is to give each channel, each row of S, its own retention factor. That is the KDA idea at the heart of Kimi Linear.

5. Kimi Linear: a separate retention factor for every channel

Gated DeltaNet fades a head's whole state with one number, every row of S by the same factor. Kimi Linear makes that control per-channel: it uses a separate retention factor for each row of S, one per channel. So each channel can fade at its own speed.

On each step, it multiplies row i of the previous state by that channel's retention factor αₜ,ᵢ, then applies the same corrective delta write using the current key and value.

That small change is powerful. Each αₜ,ᵢ is the fraction of channel i kept at this step: near 1 means a slow fade (a long effective memory), small means a fast fade (quick-cycling scratch space), all learned, all per token. Instead of one retention knob per head, the model has a whole bank of them, and can hold some learned dimensions of the state far longer than others, something a single scalar can't express.

Figure 8. Gated DeltaNet applies one decay to a head's whole state (left). Kimi Linear promotes that decay to a per-key-channel vector (right), so each channel fades at its own learned rate (α is the fraction of a channel kept per step).

This per-channel gated delta update is the KDA module (Kimi Delta Attention), introduced in the Kimi Linear report. It keeps a fixed-size recurrent state that writes correctively and forgets channel by channel, and never grows with sequence length.

Kimi Linear is the larger architecture around it: mostly KDA layers, interleaved with occasional global-attention layers, so cheap recurrent memory is punctuated by full attention that can reach across the whole context. Kimi K3 inherits this hybrid and modifies KDA's implementation: it bounds the decay parameterization so low-precision (BF16) computation stays stable, and swaps Kimi Linear's low-rank output gate for an input-dependent full-rank one.

Where this leaves us

Trace the path from Block 2:

Linear attention gave a fixed-size memory, but it could only add, so associations interfered.
DeltaNet made writes corrective: move what the current key retrieves toward the target instead of piling on (though similar keys can still interfere).
Gated DeltaNet added forgetting: one retention factor per head.
KDA (from Kimi Linear) made forgetting fine-grained: a separate retention factor per key channel.

That is a genuinely capable fixed-size memory. But it is still a summary: everything lives compressed inside S. For most tokens that's exactly right: cheap, constant in size, and recency-aware. Yet some tasks need direct, token-level access to something far back in the context, which is what a full-attention layer provides. Kimi Linear already combines the two: most layers use KDA to maintain cheap, fixed-size running memory, while periodic global-attention layers can attend directly to earlier tokens across the context. Kimi K3 inherits that hybrid and scales it into a 2.8-trillion-parameter frontier model, adding sparse experts and cross-layer retrieval along the way.

6. Kimi K3: put each mechanism where it matters

We finally have the pieces. Kimi K3 is a 2.8-trillion-parameter Mixture-of-Experts model with 104 billion parameters active per token and a context window of up to one million tokens. But the interesting part isn't the headline number; it's how K3 spends that capacity. K3 doesn't pick one memory mechanism. It combines four architectural mechanisms, each shaping information flow along a different axis of the network.

A hybrid backbone: cheap memory, occasional full attention

Most of K3's layers are KDA, the fixed-size recurrent memory from Block 5, a recurrence whose per-token decode cost doesn't grow with context length. But every fourth layer uses Gated Multi-head Latent Attention (Gated MLA): global attention that can attend directly to any earlier token in the causal context. The pattern is three KDA layers to one Gated MLA layer, repeated through the backbone (mostly cheap recurrent memory, punctuated by periodic full attention), with one extra Gated MLA layer closing the stack, so the final layer is always global. Stable LatentMoE and Attention Residuals (AttnRes) add the width-side and depth-side mechanisms.

Figure 9. A Kimi K3 block interleaves three KDA layers with one Gated MLA layer (a 3:1 ratio), each paired with a Stable LatentMoE feed-forward network. The block repeats through the backbone, with one extra Gated MLA layer closing it. Attention Residuals (AttnRes) let each layer select from the token embedding and compact summaries formed from earlier layers.

Why mix them? KDA is cheap and recency-aware but compresses everything into a fixed state; MLA keeps a genuine, addressable view of the whole sequence but its cache grows with length. Interleaving them 3:1 buys most of the efficiency of recurrence with periodic doses of true global reach. Two details make the MLA layers earn their keep:

Compressed KV (latent attention). Full attention's KV cache is the villain from Block 1. MLA, introduced in DeepSeek-V2, caches one small latent vector per past token (and reconstructs the content keys and values when attention runs), so its cache is far smaller than storing full per-head K/V, though it still grows linearly with context length.
No positional encoding (NoPE). K3 applies no explicit positional encoding to MLA's queries or keys. Instead it relies on the interleaved KDA layers for position- and recency-aware mixing, while MLA handles unrestricted global content interaction.
An output gate. A learned per-token gate independently modulates the channels read from global attention; that gate is the "Gated" in Gated MLA.

Sparse experts: 2.8T parameters, 104B active per token

Every K3 layer performs two different jobs. First, KDA or Gated MLA lets tokens gather information from the sequence. Stable LatentMoE then transforms each token's updated representation without moving information between token positions. It decides which specialized feed-forward networks, called experts, should process that token.

Each Stable LatentMoE layer has its own router and its own pool of experts. When a token reaches the layer, the router examines its current representation and selects 16 of 896 routed experts. Two shared experts also process every token. The selected experts' results and the shared experts' results are combined into one updated token representation, which continues to the next layer. That next layer has another router and makes a new decision, so the same token may use different experts at different depths. The model is therefore not splitting the sequence into permanent streams; it is choosing a temporary team of experts at each layer.

This is only a high-level view of Stable LatentMoE; if readers are interested, I can unpack its routing, compressed expert representation, and stability mechanisms in a dedicated follow-up.

Retrieval across depth: Attention Residuals

A normal residual connection carries earlier information forward by adding every layer's update to one running representation. Nothing is deliberatelydiscarded, but information from different depths becomes blended together, so a later layer cannot directly choose how much to reuse from each earlier stage.

Attention Residuals (AttnRes) preserve several summaries from different depths. In Kimi K3, a layer can draw from the token's original embedding, summaries formed by earlier groups of layers, and the running summary from earlier layers in its current group. It assigns each available source a weight and combines them into its next input. This is attention across depth: instead of reading different token positions, the model chooses among different stages of processing for the same token.

Figure 10. Kimi K3's four mechanisms and the axis each one scales: memory along the sequence (KDA), global retrieval along the sequence (Gated MLA), retrieval along depth (AttnRes), and routing along width (Stable LatentMoE).

The point

Line the four up and the design philosophy is clear. Kimi K3 adds structure along every axis of the network (sequence for KDA memory and MLA retrieval, depth for AttnRes, and width for MoE routing) rather than simply making one big dense stack larger. It routes information and computation differently along sequence, depth, and width, placing capacity where the model can use it to remember, retrieve, route, and forget.

Conclusion: architecture as memory management

Step back over the whole path (GPT-2, linear attention, DeltaNet, Gated DeltaNet, Kimi Linear, Kimi K3) and one theme runs through all of it. The thread traced here was never only about making models bigger. It was also about giving them a smarter way to manage a limited memory:

GPT-2 kept every token in its context window, and paid with a cache that grows with length.
Linear attention fixed the size, but blurred the memory together.
DeltaNet learned to edit it; Gated DeltaNet learned to forget; Kimi Linear learned to forget selectively.
Kimi K3 combined that fast recurrent memory with periodic direct token-level retrieval, sparse expert routing, and retrieval across depth.

Any memory with fixed capacity eventually has to decide what to keep and what to throw away. Gating, decay, routing, and attention are simply the tools a model uses to make that decision. Seen that way, the jump from 124 million parameters to 2.8 trillion isn't really 22,580× more of the same. It's 22,580× the rounded total parameter count, spent far more wisely.

References

Vaswani et al. (2017). Attention Is All You Need. arXiv:1706.03762
Radford et al. (2019). Language Models are Unsupervised Multitask Learners (GPT-2). OpenAI
Katharopoulos et al. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. arXiv:2006.16236
Schlag, Irie, Schmidhuber (2021). Linear Transformers Are Secretly Fast Weight Programmers. arXiv:2102.11174
Yang et al. (2024). Parallelizing Linear Transformers with the Delta Rule over Sequence Length. arXiv:2406.06484
Yang, Kautz, Hatamizadeh (ICLR 2025). Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv:2412.06464
Dao, Gu (2024). Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality (Mamba-2). arXiv:2405.21060
Widrow, Hoff (1960). Adaptive Switching Circuits (the delta rule). WESCON Convention Record.
DeepSeek-AI (2024). DeepSeek-V2 (Multi-head Latent Attention). arXiv:2405.04434
Kimi Team (2025). Kimi Linear: An Expressive, Efficient Attention Architecture. arXiv:2510.26692
Kimi Team (2026). Kimi K3: Open Frontier Intelligence. arXiv:2607.24653
Ali Taha. 22,580: GPT-2 to Kimi K3, explained. Baseten blog

How Well Can OCR Read Doctor Handwriting in 2026?

Bato — Tue, 07 Apr 2026 07:18:06 +0000

Benchmarking four open-source OCR engines on 5,578 handwritten medical prescriptions

Key Takeaways

PP-OCRv5 (5M parameters) and GLM-OCR (0.9B parameters) both achieve 20%+ exact match on handwritten prescriptions, a 10x jump over Tesseract and EasyOCR

GLM-OCR leads on character accuracy (CER 0.328), while PP-OCRv5 leads on word accuracy (WER 0.789)

A 5M-parameter model trained on curated data rivals a 900M-parameter vision-language model

Neither engine is clinically deployable yet: even the best gets only 1 in 3 words exactly right

Last month I spent some time squinting at prescription scans, trying to figure out if a doctor wrote Amoxicillin or Amitriptyline. I got it wrong twice. That got me wondering: how would today's OCR engines handle this?

The stakes are real. Medication errors injure approximately 1.3 million people annually in the United States alone and cost an estimated $42 billion globally (WHO, 2017). Illegible handwriting is a well-documented contributor: 35.7% of handwritten prescriptions contain errors, compared to just 2.5% of electronic ones (Albarrak et al., 2014).

A study of 4,183 prescriptions found that 10.21% of them are illegible, and 19.39% are barely legible (Albalushi et al., 2023). The global OCR market is projected to reach $32.9 billion by 2030, growing at a 14.8% CAGR (Grand View Research, 2025). Healthcare is one of the fastest-growing verticals driving that growth. But I couldn't find a public benchmark comparing today's open-source OCR engines on handwritten medical text.

So I ran one myself. I tested four engines on 5,578 handwritten prescription word images, and the results surprised me.

Who Are the Four Contenders?

These four engines span three generations of OCR thinking, from traditional pattern matching to specialized deep learning to generative vision-language models.

Tesseract: The Veteran

Google-backed and over 18 years old, Tesseract is the default OCR engine for a generation of developers. It uses an LSTM-based architecture designed primarily for printed text. Stable, well-documented, and runs everywhere, but handwritten cursive is not its strength.

EasyOCR: The Accessible One

Built on a CRNN (Convolutional Recurrent Neural Network) architecture with roughly 10 million parameters, EasyOCR's selling point is simplicity: pip install easyocr and you're recognizing text in 80+ languages. It uses deep learning but remains a traditional detection-recognition pipeline.

PP-OCRv5: The Data-Centric Specialist

Baidu's latest, with just 5 million parameters. PP-OCRv5 uses an SVTR_LCNet architecture with a Guided Training of CTC (GTC) strategy. The real innovation isn't the architecture, though. It's the training data.

The PP-OCRv5 paper (Cui et al., 2026) shows that data quality trumps model scale. They curated 22.6 million training samples by filtering along three dimensions. First, difficulty: they use model confidence as a proxy and found that samples in the [0.95, 0.97] range hit a sweet spot, hard enough to teach the model something new, but not so hard that the labels are unreliable. Second, accuracy: they cross-check predictions against labels to weed out mislabeled samples. Third, diversity: they cluster training images into 1,000 visual groups using CLIP embeddings and ensure each cluster is represented. Together, these filters yielded 2-3x improvements in handwritten recognition from v3 to v5 without changing the model architecture.

GLM-OCR: The Compact Vision-Language Model

From Zhipu AI and Tsinghua University, GLM-OCR takes a fundamentally different approach. It's a 0.9-billion-parameter multimodal model combining a 0.4B CogViT vision encoder with a 0.5B GLM language decoder (Duan et al., 2026). Rather than traditional CTC or attention-based sequence recognition, it generates text autoregressively, like a language model that reads images.

An important note: 0.9B is compact for a vision-language model. For comparison, Qwen3-VL has 235 billion parameters and GPT-4o is even larger. GLM-OCR was designed for efficiency, using Multi-Token Prediction (MTP) to generate approximately 5.2 tokens per decoding step, yielding a roughly 50% throughput improvement over standard autoregressive generation. It's trained through a 4-stage pipeline that includes supervised fine-tuning and GRPO reinforcement learning.

These four engines represent a clear spectrum: traditional (Tesseract), specialized deep learning (EasyOCR, PP-OCRv5), and generative VLM (GLM-OCR).

What Makes the RxHandBD Dataset So Hard?

We use RxHandBD (Shovon et al., Mendeley Data), a dataset of 5,578 cropped word images extracted from handwritten medical prescriptions written by doctors in Bangladesh. Each image contains a single word with a corresponding ground-truth label.

This dataset is hard for four reasons:

Doctor's handwriting. Enough said.
Medical terminology. Drug names like "Amoxicillin" and "Metformin" alongside dosage notations like "5% dns" and "1+0+1."
Mixed language. English medical terms interspersed with Bangla script.
Inconsistent quality. Varying paper backgrounds, pen types, and image capture conditions.

Here are a few samples to give you a sense of the difficulty range:

From top to bottom: a drug name both modern engines nail ("Ronem"), cases where only one engine succeeds ("Vineet" and "Eylox"), and a close call where GLM-OCR gets closest but still misses ("Ambrox").

One important methodological note: these are pre-cropped word-level images. We're isolating the recognition half of the OCR pipeline, not testing detection (locating text regions on a full page). This means our results reflect recognition accuracy only. Real-world performance also depends on how well each engine detects text regions before recognizing them.

How Did We Run the Benchmark?

Metrics

We evaluate using four complementary metrics:

CER (Character Error Rate): Edit distance between predicted and reference strings, normalized by reference length. If the label says "Amoxicillin" (11 characters) and the OCR outputs "Amoxicilin" (1 deletion), the CER is 1/11 = 0.09. Lower is better.
WER (Word Error Rate): Same concept at the word level. Any mistake in a word counts the entire word as wrong. Lower is better.
Exact Match Rate: The strictest metric. Did the OCR output match the ground truth character-for-character after normalization? Higher is better.
Latency: Wall-clock milliseconds per image, measuring practical throughput.

Setup

All engines run on an Apple Silicon Mac (CPU only). Single run. Default configurations for each engine with no fine-tuning or domain-specific adjustments. PP-OCRv5 runs its full detection + recognition pipeline even on pre-cropped images. GLM-OCR processes each image with the prompt "Text Recognition:".

What Do the Numbers Say?

GLM-OCR achieves the lowest character error rate at 0.328, while PP-OCRv5 leads on word-level accuracy with a WER of 0.789. Both modern engines dramatically outperform the older generation, with exact match rates 8-13x higher than Tesseract or EasyOCR.

Engine	Type	Parameters	CER (lower=better)	WER (lower=better)	Exact Match (higher=better)	Latency (ms/img)
Tesseract	LSTM	n/a	0.785	1.043	2.5%	49
EasyOCR	CRNN	~10M	0.695	1.074	2.6%	14
PP-OCRv5	SVTR+CTC	5M	0.477	0.789	21.4%	103
GLM-OCR	VLM	0.9B	0.328	0.801	32.6%	141

Let me walk through what these numbers actually mean.

Finding 1: Tesseract and EasyOCR Barely Function on Handwriting

Both Tesseract and EasyOCR achieve under 3% exact match, meaning they get fewer than 1 in 40 words perfectly right. Their WER exceeds 1.0, which means on average they produce more errors than there are words. For practical purposes, these engines are unusable on handwritten medical text.

This isn't a knock on either project. Tesseract's LSTM architecture was optimized for printed text, and EasyOCR's CRNN similarly excels on clean, well-formatted inputs. Handwritten medical cursive is simply a different problem.

Finding 2: The Modern Engines Are a Generational Leap

PP-OCRv5 and GLM-OCR both break the 20% exact match barrier, a qualitative jump from the sub-3% performance of the older engines. The gap between "old" and "new" (a roughly 10x improvement in exact match) is far larger than the gap between PP-OCRv5 and GLM-OCR themselves.

Finding 3: Why Does GLM-OCR Win on Characters but Lose on Words?

This is the most interesting finding. GLM-OCR achieves a CER of 0.328, which is 31% lower than PP-OCRv5's 0.477. At the character level, the vision-language approach genuinely helps. GLM-OCR can use its language decoder's knowledge of likely character sequences to infer partially visible characters.

But PP-OCRv5 edges ahead on WER: 0.789 vs 0.801. It makes fewer word-level mistakes.

Why the divergence? My hypothesis: GLM-OCR's autoregressive generation occasionally produces subtle extra tokens or formatting variations. The GLM-OCR paper itself acknowledges "minor stochastic variation in formatting behaviors, particularly in line breaks and whitespace handling" (Duan et al., 2026). These small artifacts barely affect CER but can flip a word from "correct" to "incorrect" in WER/exact-match scoring.

The practical takeaway: which engine is "better" depends on your error metric. If you care about getting as close as possible character-by-character (for downstream spell-correction, for example), GLM-OCR wins. If you need clean word-level outputs with minimal postprocessing, PP-OCRv5 has the edge.

Finding 4: What Are the Real Deployment Trade-offs?

Both engines are fast enough for practical use. PP-OCRv5 processes images at 103ms each and GLM-OCR at 141ms. GLM-OCR's Multi-Token Prediction (generating roughly 5.2 tokens per step instead of one) keeps its inference speed competitive despite the larger model.

The bigger difference is model size and memory footprint. PP-OCRv5's 5M parameters take up roughly 20MB on disk, small enough for a Raspberry Pi or embedded device. GLM-OCR's 0.9B parameters need around 1.8GB at FP16 (less with quantization). Both run on CPU without a GPU, as our Apple Silicon benchmark shows, but GLM-OCR consumes significantly more RAM. The GLM-OCR paper notes that the model "enables deployment in both large-scale and resource-constrained edge scenarios" and supports frameworks like Ollama for local inference (Duan et al., 2026). For deploying across thousands of low-spec workstations, PP-OCRv5's small footprint is a clear advantage. For a centralized server or any machine with a few GB of RAM to spare, GLM-OCR is equally practical.

What Do the Predictions Actually Look Like?

Numbers tell one story. Seeing the actual predictions tells another. Here are four representative samples from the benchmark, hand-picked to illustrate each finding above.

GLM-OCR Nails a Complex Drug Name

"Ceftriaxone" (a common antibiotic), 11 characters long. Tesseract reads "Wau" and EasyOCR produces "@@FNRH," both useless. PP-OCRv5 gets close with "CEFRAXONE" (CER=0.18, missing the 'ti'). Only GLM-OCR reads it perfectly. On long drug names, the language decoder's knowledge of likely character sequences gives it a real edge.

When PP-OCRv5 Is Closer and GLM-OCR Hallucinates

"Zolfin" (a proton pump inhibitor). PP-OCRv5 reads "zolfiu" (CER=0.17, just the last character wrong). GLM-OCR outputs "2016'u" (CER=1.0), completely misreading the word as a number. This is the flip side of the VLM approach: when the handwriting doesn't match patterns in the training data, the language model can steer the output in the wrong direction entirely.

GLM-OCR Nearly Perfect

"Pantonix" (a pantoprazole brand). GLM-OCR outputs "Pantomix" (CER=0.125, one character off). PP-OCRv5 gets "Pantomy" (CER=0.375). Both are close, but GLM-OCR is three times more accurate by CER. Neither gets an exact match, which is the pattern behind Finding 3: GLM-OCR consistently gets closer character-by-character, even when neither engine gets the word exactly right.

When the VLM Goes Off the Rails

"Fexo" (fexofenadine, an antihistamine). EasyOCR reads "Texo" (CER=0.25) and PP-OCRv5 reads "-exo" (CER=0.25), both reasonable attempts. GLM-OCR outputs LaTeX math notation: $ \sqrt{e} x_{0} $ (CER=4.75). This is a rare but real failure mode of generative VLMs: the model interprets the handwriting as a math expression instead of text. It produced more characters than the ground truth has, which is why CER exceeds 1.0.

What Are the Limitations?

Single dataset, single run. RxHandBD is one dataset of Bangladeshi prescriptions. US, European, or East Asian handwriting styles may produce different rankings. We don't have confidence intervals from multiple runs.
Word-level only. We tested recognition on pre-cropped word images, not full-page detection + recognition. Real-world performance depends on the complete pipeline.
CPU-only. All engines ran on CPU. GPU acceleration could significantly change the latency picture, particularly for GLM-OCR.
Default configs. No engine was fine-tuned on medical data. Domain-specific adaptation could improve all of them.
No clinical validation. OCR accuracy and clinical safety are different things. A 32.6% exact match rate is impressive for research, but not nearly sufficient for automated prescription processing without human review.

Conclusion

Modern OCR has made a genuine leap on handwritten medical text. PP-OCRv5 (5M parameters, best word-level accuracy) and GLM-OCR (0.9B parameters, best character-level accuracy) both dramatically outperform Tesseract and EasyOCR.

The two champions represent fundamentally different design philosophies: a data-centric specialized pipeline vs. a compact vision-language model. Yet they arrive at remarkably similar performance levels. Both are open-source and practically deployable.

For practitioners building healthcare OCR systems: these two engines deserve serious evaluation. Start with your specific error tolerance, hardware constraints, and whether you need word-level recognition or full-page document understanding.

For researchers: this is a domain with high clinical impact and, as these results show, plenty of room for improvement. Even the best engine here gets only 1 in 3 words exactly right on doctor handwriting. There's real work left to do.

What datasets or engines should I test next? Let me know in the comments.

References

World Health Organization. "WHO launches global effort to halve medication-related errors in 5 years." March 29, 2017. https://www.who.int/news/item/29-03-2017-who-launches-global-effort-to-halve-medication-related-errors-in-5-years
Albarrak AI, Al Rashidi EA, Fatani RK, Al Ageel SI, Mohammed R. "Assessment of legibility and completeness of handwritten and electronic prescriptions." Saudi Pharmaceutical Journal, 2014. https://pmc.ncbi.nlm.nih.gov/articles/PMC4281619/
Albalushi AK, et al. "Assessment of Legibility of Handwritten Prescriptions and Adherence to W.H.O. Prescription Writing Guidelines." J. of Pharmaceutical Research International, 2023. https://pmc.ncbi.nlm.nih.gov/articles/PMC10686667/
Cui C, Zhang Y, Sun T, et al. "PP-OCRv5: A Specialized 5M-Parameter Model Rivaling Billion-Parameter Vision-Language Models on OCR Tasks." arXiv:2603.24373, March 2026. https://arxiv.org/abs/2603.24373
Duan S, Xue Y, Wang W, et al. "GLM-OCR Technical Report." arXiv:2603.10910, March 2026. https://arxiv.org/abs/2603.10910
Shovon MSH, et al. "RxHandBD: A Handwritten Prescription Recognition Dataset from Bangladesh." Mendeley Data. https://data.mendeley.com/datasets
Grand View Research. "Optical Character Recognition Market Analysis." 2025. https://www.grandviewresearch.com/industry-analysis/optical-character-recognition-market

Benchmark conducted independently. No affiliation with any OCR project. All engines evaluated using default configurations.

About the author: Botao Deng is an ML/AI engineer and researcher who builds and evaluates production models.GitHub

To the Programmer Quietly Drowning in AI Anxiety

Bato — Sun, 22 Feb 2026 07:22:04 +0000

A quiet word for those who feel like they’re falling behind

Let me guess how your morning went.

You opened your phone, scrolled through some tech feed: Twitter, Hacker News, Reddit, whatever your poison is, and within thirty seconds, you saw someone claim they built an entire SaaS product over the weekend using nothing but prompts and vibes. Then you saw a thread about a new model that makes the one you just learned obsolete. Then a CEO somewhere declared that software engineers have maybe five good years left.

You put your phone down. You picked up your coffee. And somewhere between the first sip and the second, a familiar knot tightened in your chest.

Am I falling behind? Should I be doing more? Is everything I've built going to be worthless?

Yeah. I know that feeling. I want to talk about it.

The Treadmill That Never Stops

The pace right now is genuinely absurd. It’s not just fast, it’s disorienting. Last month’s breakthrough is this month’s footnote. You barely finish a tutorial on one framework before the community has already moved on to something shinier. The vocabulary alone is exhausting: RAG, LoRA, Agents, MCP, function calling, each one demanding your attention like a toddler pulling at your sleeve.

And the showcase culture makes it worse. Every feed is a highlight reel. Everyone seems to be shipping, building, launching. Nobody posts about the afternoon they spent confused, reading the same documentation page four times. Nobody talks about the tools they tried that turned out to be useless.

It creates this illusion that there's a speeding train, and everyone is on it except you.

But Here's What I've Learned From Watching a Few Trains Go By

I've been in tech long enough to remember when the shift from classical machine learning to deep learning felt like the sky was falling. People who had spent a decade perfecting feature engineering, tuning gradient-boosted trees, building meticulous pipelines — they woke up one day and the entire conversation had moved to neural networks. A decade of expertise suddenly felt quaint.

Then deep learning itself went through its own upheavals. CNNs gave way to RNNs, then LSTMs, then attention mechanisms, then Transformers swallowed everything whole. At each turn, someone's specialty became a paragraph in a history chapter.

Then came BERT, then GPT, and suddenly pre-training plus fine-tuning was the only game in town. Another reshuffling. Another wave of existential dread.

You know what I noticed, though? The people who came through all of that, the ones who are still here and still relevant, they weren't the ones who had the best grip on any single technology. They were the ones who had learned how to learn. They had developed a kind of peripheral vision for change: the ability to sense what mattered, what was temporary, and when to invest their energy.

That skill set doesn't expire.

Not Every Wave Deserves Your Weekend

Here's something nobody tells you when you're in the thick of it: the shelf life of most technical hype is shockingly short. The vast majority of tools, frameworks, and paradigms that seem world-ending today will be footnotes in two years. Some of them will be footnotes in six months.

This doesn't mean none of it matters. It means not all of it matters equally.

And if you try to sprint after every single thing, if you treat every new announcement as a personal emergency, you will burn out. That's not a motivational cliché. It's a mechanical fact. Human beings are not designed to sustain a permanent state of urgency.

The more useful discipline isn't relentless pursuit. It's discernment. Learning to sit with the noise long enough to separate the signal. Asking: is this a real shift in how problems get solved, or is this just a new coat of paint on an old idea? Is this changing the questions we ask, or just the tools we use to answer them?

That kind of judgment is slow to build. But it's the thing that compounds.

Why I Don't Think We're Getting Replaced

I've heard the "programmers are done" narrative enough times to have an opinion on it, so here's mine: I think it's mostly wrong, and wrong in an interesting way.

The argument assumes that programming is fundamentally about producing code, and if a machine can produce code faster, then programmers lose. But that was never quite right. The hard part of software was never typing. It was figuring out what to type. Understanding messy requirements. Navigating system constraints. Making tradeoffs that don't have clean answers. Debugging not just logic errors, but conceptual errors, the kind where the code works perfectly and the product is still wrong.

AI is extraordinary at generation. It's getting better at reasoning. But it still needs someone to point it at the right problem, to validate its output against reality, to integrate it into systems that have history and politics and technical debt. That "someone" looks a lot like an engineer to me.

And here's the irony that I think gets lost in the panic: programmers are already the people closest to this technology. We're the ones working with the models every day, feeling out their edges, learning their failure modes. The anxiety often comes from proximity, when you're standing right under a wave, it looks like it's going to crush you. But proximity is also advantage. We're not watching this from the shore. We're already in the water.

A Case for Going Slow

I want to end with something that might sound counterintuitive in an industry obsessed with speed.

It's okay to be slow.

It's okay to not have an opinion on the model that dropped yesterday. It's okay to skip a hype cycle. It's okay to spend your weekend doing something that has nothing to do with AI and not feel guilty about it.

The people who build lasting careers in technology aren't the ones who mass-produce side projects on every trending tool. They're the ones who develop taste, a quiet, hard-won instinct for what matters and what doesn't. That kind of taste doesn't come from chasing everything. It comes from watching patiently, choosing deliberately, and trusting that you don't have to catch every wave to have a good ride.

So if the anxiety has been getting to you, if you've been lying awake wondering whether your skills still matter, whether you're doing enough, whether the ground beneath you is about to give way, let me say this plainly:

You are not behind. You are in the middle of a very loud, very confusing moment. And loud, confusing moments always feel more permanent than they are.

The wave will keep moving. So will you. And at your own pace, in your own way, you'll find where you stand.

I wrote this as much for myself as for anyone else. If it landed, I'd love to hear what you're going through. I suspect a lot more of us feel this way than the highlight reels suggest.

A Beginner's Guide to Multi-Agent Systems: How AI Agents Work Together

Bato — Sat, 21 Feb 2026 18:08:55 +0000

You've probably heard the term "AI agents" thrown around a lot lately. But recently, a new idea has been taking over engineering discussions: multi-agent systems. Not one AI doing everything: but a team of AIs, each with a specific job, collaborating to tackle complex problems.

Here's a surprise: if you've ever used Claude Code to refactor a large codebase or fix a tricky bug, you've already seen a multi-agent system at work, you just might not have known it.

If that sounds complicated, don't worry. By the end of this guide, you'll understand what multi-agent systems are, why they matter, and how to build a simple one yourself (no PhD required).

First: What Even Is an "Agent"?

Before we go multi, let's make sure we're clear on what a single agent is.

A traditional LLM (like GPT or Claude) takes input and produces output — one shot, done. An agent goes further: it can reason, use tools, and take actions in a loop until a goal is completed.

Think of it this way:

LLM: "Here's a summary of that article."
Agent: "I'll search the web for that article, read it, cross-check it with two other sources, and then give you a summary with citations."

Agents typically follow a loop:

Observe → Think → Act → Observe again → ...

A common implementation looks roughly like this:

def run_agent(goal: str, tools: list) -> str:
    messages = [{"role": "user", "content": goal}]

    while True:
        response = llm.chat(messages, tools=tools)

        if response.is_final_answer:
            return response.content

        # The LLM decided to use a tool
        tool_result = execute_tool(response.tool_call)
        messages.append({"role": "tool", "content": tool_result})

Simple enough. So why do we need multiple agents?

The Problem With One Agent Doing Everything

Imagine you ask a single agent to:

"Research our top 3 competitors, write a market analysis report, and then draft 5 LinkedIn posts based on it."

That's three very different jobs: researcher, analyst, copywriter. Cramming all of that into one agent creates real problems:

Context window overload — Long tasks fill up the LLM's memory fast, causing it to "forget" earlier steps.
Lack of specialization — An agent trying to do everything tends to do nothing particularly well.
Hard to debug — When something goes wrong, you don't know which "part" failed.
No parallelism — One agent does things one at a time. What if subtasks could run simultaneously?

This is exactly the problem multi-agent systems solve.

You're Already Using Multi-Agent AI

Before we get to theory, let's look at a tool many developers already have in their terminal: Claude Code.

When you ask Claude Code something simple like "fix the bug on line 42", it handles it in a single pass. But ask it something more complex:"refactor this entire module, write tests, and check for regressions", and something more interesting happens under the hood.

Claude Code acts as an orchestrator. Instead of trying to hold the entire task in one context window, it breaks the work down and can spin up subagents: separate Claude instances with specific, scoped roles. One subagent might be tasked with exploring the codebase structure, another with writing the actual refactored code, and another with running the test suite and reporting results. Each subagent operates independently, does its job, and reports back.

You
 └─▶ Claude Code (Orchestrator)
        ├─▶ Subagent A: "Explore the repo and map dependencies"
        ├─▶ Subagent B: "Rewrite the module based on the map"
        └─▶ Subagent C: "Run tests and report failures"

The orchestrator then assembles the results and gives you a single coherent answer — as if one very capable developer had done it all.

This is the multi-agent pattern in action. And the same design is behind tools like Devin, OpenAI's Operator, and many of the AI-powered developer tools launching in 2025–2026. Now let's understand how it works so you can build your own.

What Is a Multi-Agent System?

A multi-agent system (MAS) is a setup where multiple AI agents work together — each with a defined role — to complete a larger task. Think of it like a software engineering team: you have a project manager, a frontend dev, a backend dev, and a QA engineer. Each is an expert in their lane, and a coordinator ties their work together.

The key building blocks are:

1. The Orchestrator (a.k.a. the "Manager Agent")

This is the brain that receives the high-level goal, breaks it into subtasks, assigns those subtasks to specialized agents, and assembles the final result. The orchestrator doesn't necessarily do the actual work — it delegates.

2. Subagents (a.k.a. "Worker Agents")

These agents handle specific, well-scoped tasks. A ResearchAgent searches the web. A WriterAgent drafts content. A CodeAgent writes and runs code. Each has its own set of tools appropriate to its role.

3. Tools

Tools are functions that agents can call — web search, code execution, API calls, database queries, file I/O. Tools are what make agents actually useful in the real world.

4. Memory

Agents need context. Memory can be:

Short-term (conversation history within a session)
Long-term (a vector database or knowledge store that persists between runs)

5. Communication

Agents pass messages to each other — typically as structured text or JSON. The orchestrator sends a task; the subagent returns a result.

Building a Simple Multi-Agent System

Let's put this into code. We'll build a small, framework-agnostic example: a two-agent system where one agent researches a topic and another writes a blog intro based on the research.

We'll use Python and the OpenAI API (you can swap this for any LLM provider, the pattern stays the same).

Setup

pip install openai

import openai
import json

client = openai.OpenAI(api_key="your-api-key-here")

def call_llm(system_prompt: str, user_message: str) -> str:
    """A simple wrapper to call an LLM with a system + user prompt."""
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": user_message},
        ]
    )
    return response.choices[0].message.content

The Research Agent

def research_agent(topic: str) -> str:
    """
    A specialized agent whose only job is to gather key facts about a topic.
    In a real system, this agent would have web search tools.
    For simplicity, we're having the LLM draw on its training knowledge.
    """
    system_prompt = """
    You are a research assistant. Your job is to provide a concise,
    factual summary of a given topic — 5 key bullet points, nothing more.
    Focus on accuracy and relevance. Do not editorialize.
    """
    result = call_llm(system_prompt, f"Research this topic: {topic}")
    print(f"[ResearchAgent] Done. Key facts gathered.\n")
    return result

The Writer Agent

def writer_agent(topic: str, research: str) -> str:
    """
    A specialized agent whose only job is to write engaging content
    based on provided research. It doesn't search — it just writes.
    """
    system_prompt = """
    You are a skilled technical writer for a developer blog.
    Given a topic and research notes, write a compelling, friendly
    introduction paragraph (3-4 sentences) that hooks the reader.
    Write for developers, not academics.
    """
    user_message = f"""
    Topic: {topic}

    Research notes:
    {research}

    Write the intro paragraph now.
    """
    result = call_llm(system_prompt, user_message)
    print(f"[WriterAgent] Done. Intro written.\n")
    return result

The Orchestrator

def orchestrator(goal: str) -> str:
    """
    The orchestrator receives a high-level goal, breaks it into subtasks,
    delegates to specialized agents, and assembles the final output.
    """
    print(f"[Orchestrator] Goal received: '{goal}'")
    print(f"[Orchestrator] Delegating research task...\n")

    # Step 1: Extract the topic from the goal (in a real system,
    # the orchestrator would use an LLM to parse the goal)
    topic = goal  # Simplified for this example

    # Step 2: Delegate to ResearchAgent
    research_output = research_agent(topic)

    # Step 3: Delegate to WriterAgent, passing the research output
    print(f"[Orchestrator] Delegating writing task...\n")
    final_output = writer_agent(topic, research_output)

    # Step 4: Return assembled result
    print(f"[Orchestrator] All tasks complete. Returning final output.\n")
    return final_output


# Run it
result = orchestrator("The rise of multi-agent AI systems in 2025")
print("=== FINAL OUTPUT ===")
print(result)

Sample Output

[Orchestrator] Goal received: 'The rise of multi-agent AI systems in 2025'
[Orchestrator] Delegating research task...

[ResearchAgent] Done. Key facts gathered.

[Orchestrator] Delegating writing task...

[WriterAgent] Done. Intro written.

[Orchestrator] All tasks complete. Returning final output.

=== FINAL OUTPUT ===
In 2025, AI stopped being a solo act. Multi-agent systems — where
teams of specialized AI models collaborate on complex tasks — emerged
from research labs into production engineering stacks at companies like
Google, OpenAI, and Anthropic. Rather than asking one model to do
everything, developers are now designing pipelines where a "manager"
agent delegates research, writing, coding, and verification to expert
subagents. If you've been wondering what all the buzz is about, you're
in exactly the right place.

This is the core pattern. In a production system, you'd add real web search tools, error handling, retry logic, agent memory, and parallel execution — but the orchestrator → delegate → assemble structure stays the same.

Real-World Use Cases

Multi-agent systems shine whenever a task is too large, complex, or varied for a single agent. Here are three common patterns you'll see in the wild:

1. Automated research pipelines
One agent searches and gathers sources, another reads and extracts key points, a third synthesizes findings into a report. No single agent's context window gets overwhelmed.

2. AI coding assistants (like Claude Code)
This is the most accessible real-world example. Claude Code uses an orchestrator-subagent model: when given a complex task, the main agent breaks it into subtasks and delegates — one subagent explores the codebase, one writes or modifies code, one runs shell commands and tests. Each subagent has a narrow, well-defined job. This same pattern powers tools like Devin and SWE-agent.

3. Customer support automation
An IntentAgent classifies the user's issue, a KnowledgeAgent retrieves the relevant documentation, and a ResponseAgent drafts the reply. Each agent is small, fast, and easy to tune independently.

Common Pitfalls to Avoid

Giving agents too much responsibility. The whole point of multi-agent systems is specialization. If your ResearchAgent is also writing and formatting the output, it's not really specialized.

Forgetting error handling between agents. What happens if the research agent returns nothing? Your writer agent will hallucinate. Always validate the output of one agent before passing it to the next.

Ignoring cost and latency. Each agent call costs money and time. More agents ≠ better results. Start with the minimum number of agents needed and add more only when you hit a real bottleneck.

No logging or tracing. In a chain of agents, debugging is hard without visibility. Add logs at every handoff (like the print statements in our example), and consider tools like LangSmith or Langfuse for production tracing.

Where to Go From Here

You now understand the fundamentals. Here are some good next steps depending on where you want to go:

Try LangGraph if you want a production-grade framework for building stateful, graph-based agent workflows with built-in support for cycles and conditional edges.
Try Google's Agent Development Kit (ADK) if you want Google's official framework — it was just announced and has great tooling for building hierarchical agent systems.
Try OpenAI's Agents SDK if you're already in the OpenAI ecosystem and want handoffs and tool-calling built in out of the box.
Read "Patterns for Building LLM-based Systems" by Eugene Yan — one of the best practical overviews of agent design patterns available.

Wrapping Up

Multi-agent systems aren't magic, and they're not just hype either. They're a practical engineering pattern for solving problems that are genuinely hard for a single AI to handle — tasks that are too long, too complex, or too diverse.

The pattern is simple: break down the goal → assign specialized agents → orchestrate the results. Start small, keep your agents focused, and add complexity only when you need it.

The era of AI teamwork is just getting started, and now you know how to build your own team.

Did this help? Drop a comment with what you're building — I'd love to hear what multi-agent use cases you're exploring.

Pandas 3.0's PyArrow String Revolution: A Deep Dive into Memory and Performance

Bato — Mon, 16 Feb 2026 07:50:49 +0000

Introduction

Pandas 3.0 made a game-changing decision: PyArrow-backed strings are now the default. Instead of storing strings as Python objects (the old object dtype), pandas now uses Apache Arrow's columnar format with the new string[pyarrow] dtype.

But here's the question that matters: How much does this new string dtype actually improve performance and memory usage in real-world scenarios?

To find out, I ran comprehensive benchmarks across diverse datasets and common string operations. The results? 51.8% memory savings on average, with operations running 2-27x faster.

This isn't a theoretical improvement, it's a fundamental shift in how pandas handles string data.

The Results: Summary Dashboard

Let me start with the headline numbers, then we'll dive into how I got them.

The Four Key Metrics

1. 51.8% Memory Savings

Across all test datasets, the new PyArrow string dtype used half the memory of the old object dtype. This isn't a marginal improvement, it's transformative for memory-constrained environments.

2. 6.17x Average Operation Speedup

String operations aren't just more memory-efficient: they're dramatically faster. On average, operations like str.lower(), str.contains(), and str.len() run 6x faster with PyArrow strings.

Some operations are even more impressive:

str.len(): 27x faster
str.startswith(): 16x faster
str.endswith(): 15x faster

3. 889 MB Total Memory Saved

Across our test datasets (totaling 645 MB on disk), we saved nearly 1 GB of RAM in memory. For a real data pipeline processing dozens of datasets, this compounds quickly.

4. Memory Overhead: The Game Changer

The bottom chart reveals something crucial about how pandas handles strings:

Old string dtype (object):

CSV files on disk: 645 MB
Loaded into pandas: 1,714 MB
Memory overhead: 165.7% (more than doubles!)

New string dtype (PyArrow):

CSV files on disk: 645 MB
Loaded into pandas: 825 MB
Memory overhead: 27.9% (minimal overhead)

What does this mean?

When pandas reads a CSV file, it doesn't just store the raw bytes: it creates in-memory data structures for fast operations. The old object dtype was incredibly inefficient, essentially duplicating string data multiple times. The new PyArrow string dtype keeps overhead minimal with a smarter memory layout.

This is the difference between pandas 2's Python-object approach and pandas 3's columnar Arrow approach.

The Methodology: Why 5 Different Datasets?

Now that you've seen the results, let me explain how I tested this. Real-world data comes in many shapes and sizes. A single benchmark on one type of data wouldn't tell the whole story.

That's why I created 5 distinct datasets, each representing common patterns you'll encounter in production:

1. Low Cardinality Dataset (1M rows)

What it is: Repeated categorical values like product categories, status codes, regions, and priorities.

Why it matters: This is typical of business data - think order statuses, customer segments, or department codes. The same values repeat millions of times.

Example columns:

category: "Electronics", "Clothing", "Food" (10 unique values)
status: "pending", "completed", "failed" (4 unique values)

2. High Cardinality Dataset (1M rows)

What it is: Mostly unique strings like user IDs, email addresses, and session tokens.

Why it matters: When every row is different (like customer emails or transaction IDs), pandas can't use simple optimizations. This tests worst-case scenarios.

Example columns:

user_id: "USER_00000001", "USER_00000002"... (1M unique)
email: "user123@example45.com" (1M unique)

3. Mixed String Lengths Dataset (1M rows)

What it is: A combination of short codes (2-5 chars), medium names (20-50 chars), and long descriptions (100-300 chars).

Why it matters: Real data isn't uniform. You might have product codes next to customer addresses next to order notes. This tests how pandas handles variable-length strings.

4. Dataset With Nulls (1M rows)

What it is: Data with missing values (10-33% nulls in different columns).

Why it matters: Messy data is reality. How does pandas 3.0 handle missing string data compared to pandas 2?

5. Large Dataset (10M rows)

What it is: A scaled-up version to test performance at scale.

Why it matters: Memory savings that look good at 1M rows might behave differently at 10M rows. This validates the findings scale linearly.

Memory Savings by Dataset Type

The memory savings from PyArrow strings vary significantly by dataset characteristics:

Best Case: Low Cardinality Data (-71.6%)

When data has repeated values (like categories), PyArrow strings shine:

Object dtype: 219 MB
PyArrow string dtype: 62 MB
Savings: 71.6%

Worst Case: Mixed String Lengths (-30.6%)

Variable-length strings see smaller (but still significant) savings:

Object dtype: 383 MB
PyArrow string dtype: 266 MB
Savings: 30.6%

The Pattern

Notice how savings correlate with data characteristics:

Repeated values (low cardinality) → Best savings (64-72%)
Unique values (high cardinality) → Good savings (53-55%)
Variable length (mixed sizes) → Moderate savings (31%)

Takeaway: PyArrow strings help everywhere, but they're especially powerful for categorical-like data.

Performance: Operation-Specific Speedups

This heatmap shows how much faster PyArrow strings are compared to object dtype for common string operations (values > 1.0 mean PyArrow is faster).

The Fastest Operations

str.len(): 10-27x faster
str.startswith() and str.endswith(): 11-18x faster
str.contains(): 3-5x faster
str.split(): 1-8x faster

The Pattern

Read operations (like len(), startswith()) → Massive speedups (10-27x)

These operations just examine existing data without modification

Transform operations (like replace(), split()) → Good speedups (2-5x)

These operations create new data, which limits the performance gains

The Trade-off: CSV Loading Time

There's no such thing as a free lunch. While PyArrow strings save memory and run operations faster, loading CSV files is 9%-61% slower.

Why the Slowdown?

When pandas reads a CSV with PyArrow strings enabled:

It parses the text (same as before)
It converts strings to PyArrow's columnar format (extra step)
This conversion involves building dictionary encodings and optimized memory structures

Pandas is doing more work upfront to enable better performance downstream.

Real-world impact: On our 10M row dataset, the difference is 1.63s vs 2.02s, an extra 0.4 seconds for 10 million rows. For many data pipelines, this upfront cost might be negligible compared to the 2-27x speedup in subsequent operations.

Pros and Cons: Should You Adopt PyArrow Strings?

Benefits of PyArrow String Dtype

Massive Memory Savings (30-72%)
Dramatically Faster String Operations (2-27x)
Minimal Memory Overhead (28% vs 166%)
Modern Data Ecosystem Integration

Trade-offs to Consider

Slower CSV Loading (9-61% slower)
- Initial data ingestion takes longer
- May impact workflows that repeatedly load small files
- The trade-off: slower start, much faster operations
Behavioral Changes
- String dtype behaves differently from object dtype in edge cases
- Need to update code that explicitly checks for object dtype
- Testing required for migration

The Recommendation

For most data workflows, PyArrow strings are a clear win. The memory and performance benefits far outweigh the trade-offs.

Consider staying with object dtype if:

You rarely work with string columns
Your datasets easily fit in memory
Load time is critical and you rarely perform string operations
You have legacy code that's deeply coupled to object dtype behavior

Definitely adopt PyArrow strings if:

You process large datasets with text data
String operations are a significant part of your workflow
Memory is a constraint in your environment
You're building production data pipelines
You work with modern data tools (Parquet, Arrow, DuckDB, etc.)

Conclusion

Our comprehensive analysis across 5 diverse datasets and 15+ string operations conclusively shows that PyArrow-backed strings deliver transformative improvements:

51.8% average memory savings across all dataset types
6.17x average operation speedup for string operations
Minimal memory overhead (28% vs 166% with Python objects)

PyArrow strings aren't just an incremental improvement, they're a fundamental reimagining of how pandas handles text data. By adopting Apache Arrow's proven columnar format, pandas has joined the modern data ecosystem while delivering massive performance and memory improvements.

For most data practitioners working with text data, the question isn't "Should I use PyArrow strings?" but rather "How quickly can I migrate?"

Questions or feedback? Feel free to open an issue or contribute to this analysis! The code we used in this analysis has been uploaded to this repo.

We All Accepted the "Python Tax.", Pandas 3.0 Just Reduced It.

Bato — Sun, 15 Feb 2026 06:51:21 +0000

I’ve been there. You have a "small" 3GB CSV file. You load it into a Pandas DataFrame on a 16GB machine, and suddenly everything freezes. You start manually chunking data, deleting columns, and praying to the OOM (Out of Memory) gods 🙃.

We’ve accepted this as the "Python Tax." We tell ourselves that object dtypes are just the price we pay for flexibility. Spoiler: They aren't. And we’ve been wasting RAM for years.

The "Object" Lie

For a decade, Pandas stored strings as NumPy objects. This was a beautiful abstraction with a dark secret: it’s incredibly inefficient. Each string is wrapped in a heavy Python object header. When you have 10 million rows, you aren’t just storing data; you’re storing a massive, fragmented mess of pointers.

The 10-Minute Upgrade That Saved 60% of My RAM

With the release of Pandas 3.0, the game changed. By default, it now uses a dedicated str type backed by PyArrow.

I ran the numbers because, honestly, I didn't believe at the first place. I kept my code exactly the same: no special flags, no engine tweaks, just a plain pd.read_csv(). Here is what happens when you stop using legacy NumPy objects:

The Results are Actually Insane:

Memory Slashing: In a mixed-type dataset of 10M rows, I saw a 53.2% drop in memory usage just by upgrading to version 3.0.
Text-Only DataFrame: In my experiment with 10M pure string rows, memory usage fell from 658 MB to 267 MB, 59.4% drop!

Pragmatism > Perfection

Is Pandas 3.0 perfect? No. But if you are working with text-heavy data, ignoring this upgrade is effectively choosing to pay for cloud resources you don't need.

What’s your weirdest pandas "Out of Memory" story?This type of error never fails to bring me back to the early days of pandas dev 😁