DEV Community: Dev Jhawar

Bigger AI models aren't always better: how a 70B model beat a 280B one by reading more

Dev Jhawar — Tue, 20 Oct 2026 19:02:31 +0000

For a few years, the way to build a better AI model was simple. Make it bigger.

Then in 2022 DeepMind trained a model called Chinchilla. It had 70 billion parameters. It beat their own older model, Gopher, which had 280 billion. A quarter of the size, better scores.

How? Chinchilla didn't have a smarter design. It just read a lot more.

I wanted to see that for myself, so I trained a pile of tiny models on my laptop. Here's what happened.

First, two words

Parameters: the adjustable numbers inside a model. "70B" means 70 billion of them. Think of it as the size of the model's brain.
Tokens: the chunks of text a model reads while training. Think of it as how much it studied.

Training costs compute, and compute is roughly parameters times tokens. (Times about 6, if you want the exact napkin version.) That means you can spend a budget two ways: a big brain that studies a little, or a smaller brain that studies a lot.

The old advice, and the correction

In 2020 a paper from OpenAI (Kaplan et al.) found that a model's error drops smoothly and predictably as you add parameters, data, or compute. Great, you can plan with that. Their advice for a fixed budget was to go for a very large model, train it on a relatively modest amount of data, and stop before it fully converges.

Two years later, the Chinchilla paper (Hoffmann et al.) re-ran the question more carefully and came to a different answer: model size and training tokens should grow together. Double the model, double the data.

To prove it, they trained Chinchilla with about the same compute budget as Gopher, but spent it differently:

	Parameters	Training tokens	MMLU score
Gopher	280B	300B	60.0%
Chinchilla	70B	1.4T	about 67.5%

(MMLU is a big multiple-choice knowledge test. The paper's abstract says 67.5%, a table inside says 67.6%. Same story either way.) Chinchilla also beat GPT-3 (175B), Jurassic-1 (178B) and Megatron-Turing NLG (530B) on a wide range of tasks.

The conclusion in the paper: a lot of the big models of the time were undertrained. Too much brain, not enough studying.

The rule of thumb people took from it: roughly 20 tokens of training data per parameter. Chinchilla itself is 1.4T / 70B, which is 20.

A calculator

That rule plus the napkin formula gives you a tiny calculator. If compute is 6 × N × D and D = 20 × N, then compute = 120 × N². Flip it around:

from math import sqrt

def split_budget(compute_flops, tokens_per_param=20):
    n = sqrt(compute_flops / (6 * tokens_per_param))
    return n, tokens_per_param * n

for c in [1e22, 5.76e23, 1e24]:
    n, d = split_budget(c)
    print(f"{c:.2e} FLOPs -> {n/1e9:.1f}B parameters, {d/1e12:.2f}T tokens")

1.00e+22 FLOPs -> 9.1B parameters, 0.18T tokens
5.76e+23 FLOPs -> 69.3B parameters, 1.39T tokens
1.00e+24 FLOPs -> 91.3B parameters, 1.83T tokens

The middle line is the compute number from the Chinchilla paper's table. Out comes 69B parameters and 1.4T tokens. That's Chinchilla, from one line of arithmetic.

Now let me try it on my laptop

Billions of parameters isn't something I can do. But the idea, "for a fixed budget there's a best model size", should show up at any scale. So I made a toy version.

The setup:

Data: about 12 million characters of Python source code from the standard library, with 5% held out to test on.
Models: small GPT-style transformers working on single characters, from about 3 thousand to 3.2 million parameters.
Budgets: four fixed compute budgets, from 1e11 to 3e12 FLOPs.
The trick: for each budget and each model size, I worked out how many tokens fit in the budget (tokens = compute / (6 × parameters)) and trained exactly that much. Big model, little data. Small model, lots of data. Same cost.

35 training runs. Then I measured the loss on the text the models never saw (lower is better).

Each coloured line is one budget. The star is the winner.

What I saw:

The oversized models lost, badly. At the biggest budget, the 3.2-million-parameter model got a loss of 2.61. A model about 150 times smaller, trained on far more text, got 1.75. The big one only got to read about 155 thousand tokens. That's under 0.05 tokens per parameter, a huge brain with one skim of the book.
The winner got bigger as the budget grew. Best sizes were about 7.4k, 7.4k, 10.7k and 20.6k parameters, for budgets of 1e11, 3e11, 1e12 and 3e12.
The bottom of each curve is flat. At 1e12, models of 7.4k, 10.7k and 20.6k parameters scored 2.068, 2.039 and 2.047. You don't need the exact perfect size. You need to avoid being way too big.

budget (FLOPs)	best size (params)	tokens it trained on	held-out loss
1e11	7,408	2.2M	2.477
3e11	7,408	6.7M	2.282
1e12	10,688	15.6M	2.039
3e12	20,640	24.2M	1.750

That's the same shape the Chinchilla paper describes: for a fixed budget, too-big loses, and the best model grows as the budget grows.

What I am not claiming

This is a toy, so I want to be straight about it:

I did not reproduce "20 tokens per parameter". My winners trained on hundreds or thousands of tokens per parameter. That's very different from 20. I think several things are mixed in: single-character text is a very different problem from words, my runs are tiny (from a few dozen to about twenty thousand training steps), and the position embeddings count as parameters, which is a big share of a 7k-parameter model. I didn't tune settings per model size either.
The biggest budget's winner is at the edge. At 3e12, the best model was the smallest one I could afford to test, because even smaller ones would have needed more than about 2.5 passes over my data. So its true best size is probably lower, and I can't tell from this data.
Some small models read the data more than once. For the very smallest ones I let the training text repeat up to about 2.5 times. That's harmless at these sizes, but it isn't the clean "one pass over fresh data" setup in the paper.
One run per setting. No repeats, so small differences could be noise. The big gaps are not.

So treat this as "the pattern shows up even in a toy", not "I verified Chinchilla's numbers". For the actual numbers, read the paper.

Why you should care even if you never train a model

Parameter count is a weak signal for quality. A smaller model that trained on a lot of data can beat a larger one that didn't. When you pick a model, the question "how much was it trained on?" matters as much as "how big is it?"
It's the reference point for splitting a training budget. "Just make it bigger" stopped being the default answer, and "how much data does this size need?" became a standard question.
And then it got stranger. Some labs now train small models on far more data than the 20x rule says, on purpose. That's the next post.

If you want to play with it, the whole experiment is about 100 lines of PyTorch. Change the data, the budgets or the sizes. I'd love to hear if you see the same curves, or different ones.

What it actually costs to train an LLM: GPUs, days and dollars, worked out step by step

Dev Jhawar — Tue, 20 Oct 2026 17:47:59 +0000

You've seen the headlines. "Training a big model costs tens of millions of dollars."

Okay, but why that number? Where does it come from? I wanted to count it myself, with a formula short enough to fit on a napkin, and then check the answer against something real.

Turns out you can get surprisingly close. And the places where you don't get close are the most interesting part.

Four words you need

No jargon after this, promise.

Parameters: the numbers inside the model that get adjusted during training. "8B" means 8 billion of them. Bigger usually means more capable, and more expensive.
Tokens: the little text chunks a model reads. Roughly three quarters of a word each. Training data is counted in tokens.
FLOPs: floating point operations, aka individual multiplications and additions. Training is basically an absurd number of these.
GPU-hour: one GPU running for one hour. Rent 1,000 GPUs for 10 hours and you've used 10,000 GPU-hours.

The napkin formula

Here it is:

training FLOPs  ≈  6 × parameters × tokens

That's it. Model size times data size times 6.

Where does the 6 come from? The usual explanation: for each token, the model does about 2 operations per parameter going forward (making a prediction) and about 4 going backward (figuring out how to improve). 2 + 4 = 6. It's a rule of thumb, not exact, but it's the one everybody uses.

Quick sanity check, because I don't trust a formula until it matches something:

Chinchilla, a famous 70B model trained on 1.4 trillion tokens: 6 × 70e9 × 1.4e12 = 5.9e23 FLOPs. The paper's own table says about 5.8e23.
Llama 3.1 405B, trained on 15.6 trillion tokens: 6 × 405e9 × 15.6e12 = 3.8e25. Meta's paper says 3.8e25.

Matches. Good enough for a napkin.

Turning FLOPs into hours

Now, how fast does a GPU chew through FLOPs?

An NVIDIA H100 (the chip these big runs use) has a peak of about 989 TFLOP/s in the number format used for training. That's 989 trillion operations per second. Sounds huge. It is.

But "peak" is the speed on the spec sheet, in a perfect world. Real training spends time moving data around, waiting on other GPUs, and so on. Meta said their best setup reached over 400 TFLOP/s per GPU, which is about 40% of peak. (Their paper puts utilisation at 38 to 43%.)

So:

GPU-hours = FLOPs / (speed per GPU × 3600)

Napkin vs what Meta reported

Meta published GPU-hours for their models. So I could check my napkin against the real thing.

Model	tokens	6 × N × D	napkin hours at 400 TFLOP/s	napkin hours at 200 TFLOP/s	Meta reported
Llama 3 8B	15T	7.2e23	0.5M	1.0M	1.3M
Llama 3 70B	15T	6.3e24	4.4M	8.8M	6.4M
Llama 3.1 405B	15.6T	3.8e25	26.3M	52.6M	30.8M

(I used the nominal sizes, 8B, 70B and 405B. The real counts are a bit different.)

The 405B result is fun: Meta reported 30.8M GPU-hours, and my napkin at 400 TFLOP/s said 26.3M. Not bad.

But look at the 8B. Meta reported 1.3M hours, and my napkin said 0.5M. Off by 2.6x.

So I flipped the question around. If Meta's hours are right, what speed did the GPUs actually deliver? Dividing the work by the reported hours gives:

8B: about 154 TFLOP/s, 16% of peak
70B: about 273 TFLOP/s, 28% of peak
405B: about 341 TFLOP/s, 35% of peak

The bigger the model, the better the GPUs were used. My guess, and it is a guess, is that small models don't keep a huge GPU busy as efficiently, and that the 6ND rule skips some real costs (attention, restarts, evaluation runs). I haven't dug into Meta's logs, so take that as speculation. The lesson I'd actually take: the napkin is great for order of magnitude, and the weak spot is always GPU speed. Use a range, not one number.

Okay, but dollars

GPU-hours times price per GPU-hour.

Big caveat: Meta owns its GPUs, so this isn't what Meta spent. It's what you'd pay to rent that much compute, and rental prices vary a lot. I'm using $2 to $4 per H100-hour as an assumption, not a quote.

Model	Meta's reported GPU-hours	at $2	at $4
Llama 3 8B	1.3M	$2.6M	$5.2M
Llama 3 70B	6.4M	$12.8M	$25.6M
Llama 3.1 405B	30.84M	$61.7M	$123.4M

That's just the final training run, by the way. It doesn't include failed experiments, data work, salaries, or the cost of all the smaller models you train first to figure out what to build.

"How many GPUs" is a question about time

Here's the part people miss. GPU-hours is a total. How many GPUs you need depends on how long you're willing to wait:

GPUs needed = GPU-hours / hours you can wait

The 8B run at 1.3M GPU-hours:

On 1,000 GPUs: 1,300 hours, about 54 days.
On 2,048 GPUs: about 635 hours, around 26 days.

(Those cluster sizes are my examples, not what Meta used for that model.)

For the 405B: 30.84M GPU-hours spread over 16,384 GPUs is about 1,880 hours, roughly 78 days, if every GPU worked the whole time. Real runs don't, so treat that as a rough ballpark.

Try it yourself

Here's the whole thing as a script. Change the numbers.

def train_cost(params, tokens, tflops_per_gpu, dollars_per_gpu_hour, n_gpus=None):
    flops = 6 * params * tokens
    gpu_hours = flops / (tflops_per_gpu * 1e12) / 3600
    out = {"FLOPs": f"{flops:.2e}",
           "GPU-hours": f"{gpu_hours:,.0f}",
           "cost": f"${gpu_hours * dollars_per_gpu_hour:,.0f}"}
    if n_gpus:
        out["days"] = f"{gpu_hours / n_gpus / 24:.1f} days on {n_gpus} GPUs"
    return out

# a 1B model, 20 tokens per parameter (Chinchilla's rule of thumb), one H100 at 150 TFLOP/s
print(train_cost(1e9, 20e9, 150, 2))
print(train_cost(1e9, 20e9, 150, 4))

{'FLOPs': '1.20e+20', 'GPU-hours': '222', 'cost': '$444'}
{'FLOPs': '1.20e+20', 'GPU-hours': '222', 'cost': '$889'}

So a small 1B model, on a sensible amount of data, is about $450 to $900 of rented GPU time. That's a "weekend project and a credit card" number, not a "nation-state" number. It's the 8B going to 70B going to 405B that moves you from "hobby" to "company" because the work multiplies each time.

And for fun, I measured my own laptop CPU while running small experiments: about 30 GFLOP/s on a tiny model. Same 1B model:

1.2e20 FLOPs / 3e10 FLOP/s = 4e9 seconds  ≈  127 years

So no, I won't be training that one on the couch.

What I'd take away

6 × parameters × tokens is a good napkin for training work. It reproduced both Chinchilla's and Llama 3.1 405B's published compute.
GPU speed is the shaky number. Real training reached roughly 16% to 35% of an H100's peak in Meta's reported hours. Always give a range.
Dollars depend on rent, and rent is an assumption. The reported GPU-hours are the solid part.
Doubling the model, or the data, doubles the bill. Doubling both quadruples it.

Honest limits

I used nominal model sizes, rented-GPU prices I picked myself, and Meta's reported hours as ground truth. The explanation for why the 8B ran at a lower share of peak is my guess. And the 6ND rule is a simplification: real runs have extra costs it ignores.

If you want to check my arithmetic, the numbers in the table come straight from the two Llama model cards and the Llama 3 paper. Run the script with your own model size and see what a "small" model costs. I was surprised how small.

Why Meta trained a small 8B model on 15 trillion tokens, and why that's cheaper than it sounds

Dev Jhawar — Tue, 20 Oct 2026 17:47:35 +0000

There's a rule of thumb from a famous paper: for the best result per dollar of training, feed a model about 20 tokens of text for every parameter it has.

An 8-billion-parameter model times 20 is 160 billion tokens. Meta's own blog post puts the "Chinchilla-optimal" amount for an 8B model at about 200 billion.

Llama 3 8B was trained on over 15 trillion.

That's roughly 75 to 95 times more than the rule says. Either Meta wasted a ridiculous amount of compute, or the rule is answering a different question than the one Meta was asking. I wanted to figure out which.

The rule is about the cost of building

The Chinchilla paper (2022) asked: if I have a fixed amount of compute to spend on training, what mix of model size and data gives the best model?

The answer was to grow both together. Their 70B model, trained on 1.4 trillion tokens, beat the older 280B Gopher, which only saw 300 billion tokens.

But look at what that question leaves out. A model gets trained once. Then it gets used. Maybe by millions of people, for years.

So there are two bills:

Building: about 6 × parameters × tokens FLOPs, paid once. (FLOPs are just individual multiplications and additions. Training is an absurd number of them.)
Serving: about 2 × parameters FLOPs for every token the model processes. Paid again, every time anyone uses it.

The serving bill scales with model size. A smaller model is cheaper on every single request, forever.

So here's the trade. Take a smaller model, train it on way more data than the rule suggests, and you can get a model that's as good as a bigger one. You pay more upfront. You pay less on every answer.

Putting numbers on it

The Chinchilla paper also gives a formula that predicts how good a model will be (lower loss = better) from its size and training data:

loss = 1.69 + 406.4 / N^0.34 + 410.7 / D^0.28

N is parameters, D is tokens. In plain words: there's a floor you can't beat (1.69), then a penalty for having a small model, then a penalty for not having read enough. Make N bigger and the second penalty shrinks. Feed it more data and the third one does.

Let's use it. First, what does it predict for the 8B on 15T tokens?

E, A, B, a, b = 1.69, 406.4, 410.7, 0.34, 0.28
loss = lambda N, D: E + A / N**a + B / D**b

print(loss(8e9, 15e12))        # 8B, trained on 15T tokens
print(loss(8e9, 20 * 8e9))     # 8B, at the 20-tokens-per-parameter rule

1.949
2.164

So by the formula, training the same 8B model on 15T tokens instead of 160B drops the predicted loss from 2.164 to 1.949. That's the whole reason to overtrain: the model keeps getting better, just with diminishing returns. (Meta's blog says both Llama 3 models "continued to improve log-linearly" up to 15T tokens, so that matches the idea.)

Now flip it around. What's the cheapest-to-train model that reaches that same predicted loss of 1.949?

import numpy as np

target = loss(8e9, 15e12)

def tokens_needed(N):
    gap = target - E - A / N**a
    return (B / gap) ** (1 / b) if gap > 0 else np.inf

best = min(np.logspace(9.5, 12, 4000), key=lambda N: 6 * N * tokens_needed(N))
print(best / 1e9, tokens_needed(best) / 1e12)

26.1  2.31

A 26B model trained on 2.3 trillion tokens is predicted to land in the same place. Its training bill, by the napkin formula, is 6 × 26.1e9 × 2.31e12 = 3.6e23 FLOPs.

The 8B on 15T costs 6 × 8e9 × 15e12 = 7.2e23. Twice as much to build.

So overtraining really is the more expensive way to build. If building were the whole story, Chinchilla's rule wins.

But then people use it

Total cost = building + serving. Serving costs 2 × N per token processed:

C_small, N_small = 7.2e23, 8e9
C_big,   N_big   = 3.62e23, 26.1e9

for served in [1e12, 1e13, 1e14, 1e15]:
    small = C_small + 2 * N_small * served
    big   = C_big   + 2 * N_big   * served
    print(f"{served:.0e} tokens served: big model costs {big/small:.2f}x the small one")

1e+12 tokens served: big model costs 0.56x the small one
1e+13 tokens served: big model costs 1.00x the small one
1e+14 tokens served: big model costs 2.41x the small one
1e+15 tokens served: big model costs 3.14x the small one

There's a break-even point at around 10 trillion tokens served. Below that, the bigger, cheaper-to-train model wins. Above it, the small overtrained one does, and the gap keeps widening: at 100 trillion tokens, the big one costs about 2.4 times as much in total.

For something that's going to be downloaded, fine-tuned and run by a lot of people, you'd expect to be well past that. That's the bargain: pay extra once, save on every answer.

What this means if you build agents

Agents are token-hungry. Every step re-reads a long prompt, so you process a lot more tokens than the user ever sees, which pushes you toward the "serving matters" side of that chart.

It's also a hint about why smaller models keep getting good enough for tool-calling work. A small model that was trained on a mountain of data can behave like a bigger, older model. If you're picking a model for a loop that runs thousands of times a day, size is a cost you pay every call.

Honest limits

This is a fun calculation, not a proof, so a few things to keep in mind:

I'm using a formula far outside its comfort zone. The Chinchilla loss formula was fitted on training runs much smaller than 15 trillion tokens. At least one later paper refitted the same data and got quite different constants. Treat the 26B as "somewhere in that ballpark", not an exact answer.
Loss isn't quality. Two models with the same predicted loss can still differ on real tasks.
FLOPs aren't dollars. Serving a big model also needs more GPUs just to fit it in memory, which my 2N shortcut ignores. That would tilt things further toward the small model, but I haven't modelled it.
I don't know Meta's actual reasoning. I'm not claiming this is why they did it. I'm showing that the numbers make it a reasonable bet.
I used the nominal 8B. The real parameter count is slightly different.

Run the snippets, change the model sizes, change how much you expect to serve. The break-even point moves a lot, and that's kind of the point: "optimal" depends on what you're optimising for.

The KV cache is eating your GPU. MQA and GQA are how models fight back

Dev Jhawar — Tue, 20 Oct 2026 17:16:12 +0000

Quick recap from the last post: the KV cache saves you from recomputing old tokens, and it's the reason generation is fast.

But it has a bill. And the bill is memory.

I wanted to put an actual number on it, because "the cache is big" doesn't mean much until you see the GB.

The formula

Per token, you store a Key and a Value for every layer and every head:

bytes per token = 2 x layers x kv_heads x head_dim x bytes_per_number

The 2 is K and V. fp16 is 2 bytes per number. Here it is as code:

def kv_bytes(layers, kv_heads, head_dim, tokens, nbytes=2):
    return 2 * layers * kv_heads * head_dim * nbytes * tokens

Now plug in a Llama-2 7B style model: 32 layers, 32 heads, head dim 128.

That's 512 KB for every single token. At 8k context, 4 GB. At 128k context, 64 GB. For one request. Weights not included.

If you're serving many users at once, each one has their own cache. That's the thing that actually fills your GPU, not the model.

Look at the formula again

Layers and head_dim are baked into how good the model is. Bytes per number you can shrink with quantization, but that's a different post.

The interesting variable is kv_heads.

In standard multi-head attention (MHA), every query head has its own key head and value head. 32 query heads, 32 KV heads.

But do the heads really need separate Keys and Values? Turns out, mostly no.

MQA: one KV head for everybody

Multi-Query Attention keeps all the query heads but shares a single K head and V head across all of them. 32 query heads, 1 KV head.

Memory goes down by 32x. The catch is quality: forcing every head to read from one shared set of keys and values can hurt, and training gets a bit touchier.

GQA: the compromise

Grouped-Query Attention sits in between. Split the query heads into groups, and each group shares one KV head.

Llama-3 8B has 32 query heads but only 8 KV heads. So 4 query heads share each K/V pair. That's a 4x cut in cache size, and the published results say quality stays close to full MHA.

Same model shape, three attention styles:

Style	KV heads	KB per token	8k context	128k context
MHA (Llama-2 7B)	32	512	4 GB	64 GB
GQA (Llama-3 8B)	8	128	1 GB	16 GB
MQA (hypothetical, same shape)	1	16	0.12 GB	2 GB

The MQA row is me taking the same shape with one KV head, to show the extreme. It's not a model I measured.

What changes in the code

Less than you'd think. The only real difference is that you repeat the KV heads to line up with the query heads at attention time:

# k, v: (kv_heads, t, head_dim)   q: (q_heads, 1, head_dim)
group = q_heads // kv_heads
k = np.repeat(k, group, axis=0)   # now (q_heads, t, head_dim)
v = np.repeat(v, group, axis=0)

Important detail: you store the small version in the cache, and expand it only when computing. That's where the memory saving lives. (Real implementations avoid the copy, but the idea is the same.)

Why this matters if you're building agents

You probably aren't training a model. But:

When a provider says a model supports 128k or 1M context, GQA-style tricks are a big part of why that's affordable to serve at all.
Long contexts are cheaper per token on models built this way, which shows up in latency and sometimes in price.
If you self-host, KV memory is the number that decides how many concurrent users fit on one GPU. Do this arithmetic before buying hardware.

Where i'd be careful

Those numbers are for one sequence at fp16. Batching multiplies them, quantizing the cache divides them.
"Little quality loss" comes from papers and model cards. I haven't run my own quality evals here, and you shouldn't take it from a blog post either. Check the benchmarks for the specific model you use.
The GB figures are straight arithmetic from the formula, not profiler output. Real servers add overhead.

Next up: why the same sentence can cost very different amounts depending on the tokenizer. Slightly less hardware, more wallet.

Tokenization is why your agent bill looks weird

Dev Jhawar — Tue, 20 Oct 2026 17:15:47 +0000

Models don't read text. They read integers.

Between your string and the model there's a tokenizer, and it quietly decides how much you pay, how much fits in the context window, and why your model is weirdly bad at counting letters.

I always treated it as a preprocessing detail. Then i started counting.

What a tokenizer actually does

It chops text into pieces from a fixed vocabulary, usually 30k to 200k entries, and maps each piece to an id. Common words become one token. Rare stuff gets split into smaller bits.

You can see it directly with tiktoken:

import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
for w in ("tokenization", " tokenization", "Pneumonoultramicroscopicsilicovolcanoconiosis"):
    print(repr(w), [enc.decode([t]) for t in enc.encode(w)])

'tokenization'  ['token', 'ization']
' tokenization' [' token', 'ization']
'Pneumonoultramicroscopicsilicovolcanoconiosis'
  ['P', 'ne', 'um', 'on', 'oul', 'tram', 'icro', 'sc', 'op', 'ics', 'il', 'ic', 'ov', 'ol', 'cano', 'con', 'iosis']

A common word: two pieces. A rare one: seventeen. Also notice that a leading space changes the token. Small thing, but it's why prompts with different whitespace can behave differently.

How the vocabulary gets built

Most modern ones are some flavour of BPE (byte pair encoding). The idea is almost boringly simple: start with single characters, find the most frequent adjacent pair, merge it into a new symbol, repeat.

I ran it on a tiny made-up corpus (low x5, lower x2, newest x6, widest x3):

1  merge s+t   (9x)   n e w e st
2  merge e+st  (9x)   n e w est
3  merge o+w   (7x)   l ow
4  merge l+ow  (7x)   low
5  merge w+est (6x)   n e west
6  merge n+e   (6x)   ne west

Do that for tens of thousands of merges over a huge corpus and you get a vocabulary. Whatever was common in the training text gets short, cheap tokens. Whatever wasn't, doesn't.

That last sentence is the whole post.

The families, quickly

BPE: merge frequent pairs. GPT-2 style.
Byte-level BPE: BPE over raw bytes, so there's never an "unknown" character. Anything can be encoded, it just might take many tokens. GPT-2 and the OpenAI tiktoken encodings work this way.
WordPiece: similar, but picks merges by likelihood instead of raw frequency. BERT uses it.
SentencePiece: works on raw text including spaces (no pre-splitting on whitespace), so it handles languages without spaces. T5 and many other models use it.

So I counted

Same strings, four tokenizers: GPT-2, cl100k_base (GPT-4 era), o200k_base (GPT-4o era) and T5's SentencePiece. Normalised to tokens per 100 characters so short and long samples compare.

Raw counts:

Text	chars	GPT-2	cl100k	o200k	T5
English sentence	84	18	18	18	22
Python snippet	176	83	43	44	61
JSON tool schema	610	329	160	161	183
One Hindi sentence	74	111	78	24	31
Emoji string	12	28	32	18	8

(These are single samples I picked, not a benchmark. Your text will differ. T5 and the others also have different vocabularies, so it isn't a pure "which is better" ranking.)

A few things jumped out.

English is the happy path. Everyone's about the same, because every vocabulary was built mostly on English.

Code and JSON halved between GPT-2 and the newer encodings. My guess is that newer vocabularies were trained on a lot more code, so runs of whitespace and common JSON punctuation get merged into single tokens. I didn't dig into the vocab files to confirm that.

Hindi is where it hurts. 111 tokens in GPT-2, 78 in cl100k, and 24 in o200k. Same sentence. With the older encoding it's more than 4x the cost and 4x the context used, and I'd guess quality is worse too, though I didn't measure that. Dropping to individual characters is what a byte-level tokenizer does when it has never seen your script much:

'मेरा' -> ['म', 'े', 'र', 'ा']   # one token per character/mark

Emoji are all over the place.

Why this matters for agents

Tool schemas are paid for on every call. My little 610 character JSON schema is ~160 tokens. Twenty tools like that is 3,000+ tokens of overhead on every single step of the loop. Shorter descriptions, fewer tools per call, and a stable ordering (so prompt caching works, see the KV cache post) all help.
Pretty-printed JSON costs more than compact JSON. Indentation is tokens. Try json.dumps(obj, separators=(",", ":")) on your tool results.
Count before you guess. Use the actual tokenizer for your model. Characters / 4 is a fine rule of thumb for English and a terrible one for Hindi, code, or emoji.
Non-English users pay more, in money and in context. If your product serves them, measure with their text, not yours.
Weird model failures like counting letters in "strawberry" or arithmetic on long numbers are partly tokenization: the model never saw the letters, only the chunks.

Honest limits

I tested one sample per category, with four tokenizers. It's an illustration of the effect, not a study. Also I'm only showing token counts; whether a model does well in a language depends on training data, not only on how its text is split.

If you want to try it: swap in your own strings, your own language, your own tool schemas. Ten minutes, and you will probably find something that surprises you. I did.

Why your agent gets slower and pricier with every tool call

Dev Jhawar — Tue, 20 Oct 2026 17:09:05 +0000

Okay so here's something that bugged me for a while.

You build an agent. Step 1 feels snappy. By step 15, after a few tool calls and a pile of JSON results, it's noticeably slower, and the bill is climbing in a way that doesn't feel linear.

I knew the vague answer ("the context is longer"). I didn't really get it. So i stopped reading and wrote the thing myself, in plain NumPy, to see where the time actually goes.

Short version: the culprit is a boring little thing called the KV cache, and once you see it, a lot of agent design advice suddenly makes sense.

What happens when a model writes one token

A transformer generates text one token at a time. For every new token, attention needs the Key and Value vectors of every token before it. That's how the new token "looks back".

Here's the catch. Those old Keys and Values don't change. Token 12's Key is the same whether you're generating token 13 or token 913.

So a naive loop does something silly: at every step it recomputes K and V for the whole sequence, from scratch, then throws them away. Then does it again next step.

The KV cache is just... not doing that. You compute K and V for a token once, keep them in memory, and append.

Let me show you, not tell you

Here's the toy model. 4 layers, 8 heads, 512 wide, random weights. It's not smart. It doesn't need to be, we're timing the machinery.

No cache, the full thing re-runs every step:

def forward_full(X):
    h = X
    t = h.shape[0]
    mask = np.triu(np.full((t, t), -1e9, np.float32), 1)
    for w in W:
        n = rms(h)
        q, k, v = (split(n @ w[m]) for m in "qkv")
        s = q @ k.transpose(0, 2, 1) / np.sqrt(HD) + mask
        h = h + merge(softmax(s) @ v) @ w["o"]
    return h[-1:]

With a cache, only the newest token goes through the projections:

def forward_cached(x, cache):
    h = x
    for w, c in zip(W, cache):
        n = rms(h)
        c["K"] = np.concatenate([c["K"], split(n @ w["k"])], 1)
        c["V"] = np.concatenate([c["V"], split(n @ w["v"])], 1)
        q = split(n @ w["q"])
        s = q @ c["K"].transpose(0, 2, 1) / np.sqrt(HD)
        h = h + merge(softmax(s) @ c["V"]) @ w["o"]
    return h

Same math. Before timing anything I checked both versions produce the same output over 40 steps (max difference around 1e-6). Honestly this was the step i almost skipped, and my first version of the benchmark failed it, the "no cache" path had a bug and I was about to publish a fake speedup. Check your baselines.

The numbers

I generated 1,024 tokens each way on my laptop CPU.

Without the cache, the time per token keeps climbing: about 12 ms at step 16, 174 ms at step 512, 550 ms at step 1023. With the cache it stays low, 3 ms early on and about 16 ms at the end.

Total time for all 1,024 tokens: 232 seconds vs 12.5 seconds. Roughly 18x.

Two things worth being honest about.

First, this is a toy on a CPU in NumPy. A real LLM on a GPU has different constants, and nobody should quote my 18x as if it applies to GPT-anything. The shape is what matters: one curve grows with length, the other mostly doesn't.

Second, the cached line isn't flat. It still creeps up (3 ms to 16 ms). That's because the new token still has to attend over everything in the cache. The cache removes the recomputation, not the reading.

Okay but what about agents

This is the part I actually care about.

An agent loop looks like this: model call, tool call, append the result, model call again. Every iteration sends the whole conversation back, system prompt, tool schemas, history, all of it.

If the provider has to process that whole prefix from scratch each time, you pay for it each time, in latency and in input tokens. That's the "slower and pricier with every tool call" feeling.

Prompt caching is the same idea one level up. Providers keep the KV cache for a prefix they've seen, so if your next request starts with the same prefix, they skip that work, and usually bill it at a discount.

Which has some very practical consequences:

Keep the prefix stable. A timestamp or a request ID at the top of your system prompt changes the prefix every call and kills the cache. Put the volatile stuff at the end.
Don't reorder or edit earlier messages. Rewriting history, "cleaning up" older tool results, shuffling tool definitions: all of it breaks the cached prefix from that point onward.
Append-only is your friend. New content goes on the end. That's the pattern caches are built for.
Big tool results cost you forever. A 20k-token JSON dump stays in the context for every following step. Trim it before it goes in, not after.
Check your provider's rules. Cache lifetimes, minimum prefix sizes and pricing differ. Read the docs, don't assume.

Not glamorous. But it's the difference between an agent that costs cents and one that costs dollars.

The catch: the cache itself is huge

Fine, so just cache everything? The cache lives in memory, and it grows with every token.

The formula is simple:

bytes per token = 2 (K and V) x layers x kv_heads x head_dim x bytes per number

For a Llama-2 7B style model that's 512 KB per token. At 128k context, one single request needs about 64 GB just for its cache. That's before the model weights.

That's why the next thing people ask is "how do we make this smaller?" and the answer is a neat trick with attention heads. I'll cover it in the next post.

What i'd take away

The KV cache isn't an optimization you opt into. It's why generation is usable at all.
Your agent gets slower with every step because the context it re-reads keeps growing, and cost follows.
The cheapest fix is usually boring: stable prefix, append-only history, trimmed tool results.

If you want to poke at it yourself, the whole benchmark is about 80 lines. Change the layer count or the length and watch the curves move. Tell me if you get different shapes, i'm curious.

Running 100 agents on a free tier (Groq 30 RPM, Tavily credits)

Dev Jhawar — Tue, 20 Oct 2026 16:27:27 +0000

GridMind is a spreadsheet where every cell is its own research agent. You type "Stripe" in a row, "who is the CEO?" in a column, hit Run, and an agent goes off, searches the web, and fills the cell with an answer and sources.

You can try it live at grid-mind-tan.vercel.app, and the code is on GitHub.

Cute idea. Then you do the maths.

A 10 x 10 sheet is 100 cells. Each cell needs an LLM call to plan, one or two searches, and an LLM call to answer. Groq's free tier gives me around 30 requests a minute. Tavily gives 1000 search credits a month. So one enthusiastic click on "Run all" can eat a decent chunk of the month, and the 429s start before the first row is even done.

I didn't want to pay for anything yet. So the job became: make the free tier last. It turned out to be four small things, none of them clever.

1. A limiter that sits under the limit

The first fix is the obvious one: don't send more than the provider allows. It's a sliding window, about 20 lines (resilience.py):

class RateLimiter:
    def __init__(self, rate: int, per: float = 60.0) -> None:
        self.rate, self.per = max(1, rate), per
        self._hits: deque[float] = deque()
        self._lock = asyncio.Lock()

    async def acquire(self) -> None:
        while True:
            async with self._lock:
                now = time.monotonic()
                while self._hits and now - self._hits[0] >= self.per:
                    self._hits.popleft()
                if len(self._hits) < self.rate:
                    self._hits.append(now)
                    return
                wait = self.per - (now - self._hits[0])
            await asyncio.sleep(wait + random.random() * 0.1)

Two details I care about. The limit is set to 25 for Groq, not 30. If you aim exactly at the line, clock skew and a retry or two will push you over it. And the little random sleep at the end is there so a hundred waiting workers don't all wake up on the same tick and stampede.

2. When it still fails, retry like an adult

Limiter or not, you will get a 429 or a 503 sometimes. The retry is exponential backoff with jitter, and it respects Retry-After when the server sends one. If the server tells you how long to wait, believe it:

retry_after = getattr(getattr(exc, "response", None), "headers", {}).get("retry-after")
delay = float(retry_after) if retry_after and str(retry_after).replace(".", "").isdigit() else None
await asyncio.sleep(min(cap, delay if delay is not None else random.uniform(0, base * 2**i)))

Only transient errors (408, 429, 5xx, timeouts) are retried. Hitting my own daily call budget for the LLM is not transient either, retrying it just burns time, so that one raises a separate non-retryable error.

3. Stop asking the same question twice

This one saved the most. In a sheet about companies, the cells in one row ask near-identical things about the same entity. Searching "Stripe CEO" and "Stripe chief executive" is wasted money when the results are basically the same.

So searches and page reads go through a small cache with a TTL (an hour by default). The part that matters is the in-flight de-dup: if ten cells ask for the same thing at the same moment, only the first one actually goes out, and the rest wait on its result.

if key in self._inflight:
    return await asyncio.shield(self._inflight[key])

A plain cache doesn't help here. When you hit Run all, all the identical requests start before any of them has finished, so there's nothing in the cache yet. Sharing the pending call is what fixes that. If the shared call fails, the failure is shared too, and nothing bad gets cached.

4. Spend the paid credit once, and have a floor to fall to

Tavily is good, so it gets the first query of each cell. The second query goes to DuckDuckGo, which is free. A cell costs one credit instead of two.

On top of that there's a counter that stops me at 900 credits a month, so I never hit the 1000 wall mid-run. And if Tavily errors out or the budget is gone, the search quietly falls back to DuckDuckGo. Results get a bit worse, but the sheet still fills.

DuckDuckGo gets rate-limited a lot too. For the secondary query that's fine, a miss just returns an empty list and the agent works with what it has. For the primary query, a miss is a real error.

What this doesn't solve

Being honest here:

Every limiter and counter lives in memory, per instance. Run two API instances and you've doubled your limits without meaning to. Going global means Redis counters, which I haven't done yet.
The fallback means answer quality is not constant. A cell answered from DuckDuckGo snippets can be thinner than one answered from Tavily.
I picked 25, 40 and 900 by feel and the providers' published limits, not by load testing.

None of it is fancy. A limiter, a retry, a cache that shares in-flight work, and a budget with a fallback. But together they're why a sheet of 100 agents can run on a setup that costs me nothing.

If you want to see the limiter at work, open any cell in the live demo and check its trace. Every search the agent made is listed there.

Code is here if you want to poke at it: github.com/DEVJHAWAR11/GridMind