DEV Community: Jangwook Kim

Why My Local Agent Forgot Its System Prompt — Measuring Ollama num_ctx Silent Truncation

Jangwook Kim — Sun, 28 Jun 2026 06:50:17 +0000

A few days ago, a meeting-notes summarizer I run locally started misbehaving. Short transcripts were fine, but feed it a long transcript and it ignored the instruction I'd written at the top ("answer in JSON only") and rambled in plain prose. My first guess was that the model was just dumb. What nagged me was that the same model obeyed the instruction perfectly on short inputs. So maybe the model hadn't gotten dumber. Maybe it had never seen my instruction at all.

In a previous post that decomposed prefill and generation cost, I measured how a longer context delays the first token. That was about speed. What I suspected this time was something else. Past a certain length, maybe it isn't speed that suffers but the content that vanishes. So I measured it.

Hiding a secret at the front, then growing the length

The method is a small twist on needle-in-haystack. Drop a secret code on the first line of the prompt (the head). Below it, lay down a long stretch of filler text, like a meeting transcript, to inflate the token count. Then at the very end, ask "what was the secret code written at the top?" If the model answers it correctly, it saw the head. If it can't, the head is gone.

The key is to keep the same prompt and only change num_ctx. If recall breaks while the input is identical, the culprit isn't the model. It's the context window setting.

import json, urllib.request

SECRET = "ALPHA-7723-ZULU"
HEAD = (f"IMPORTANT: a secret code is hidden somewhere in this document. "
        f"The secret code is {SECRET}.\nRead the notes below, but answer only the final question.\n\n")
FILLER = ("The meeting covered the quarterly roadmap, deploy schedule, on-call rotation, and cost cuts. "
          "Each team shared progress and reordered next sprint's priorities.\n")
Q = "\n\nQuestion: what is the secret code written at the top of this document? Answer with the code only."

def ask(num_ctx, n_filler):
    prompt = HEAD + (FILLER * n_filler) + Q
    body = {"model": "melavisions/gemma4:latest", "prompt": prompt,
            "stream": False,
            "options": {"num_predict": 40, "temperature": 0, "num_ctx": num_ctx}}
    req = urllib.request.Request("http://localhost:11434/api/generate",
            data=json.dumps(body).encode(), headers={"Content-Type": "application/json"})
    d = json.load(urllib.request.urlopen(req, timeout=600))
    return d["prompt_eval_count"], SECRET in d["response"], d["response"].strip()

I used the gemma4:latest (3.2B, Q4_K_M) quantized build. It's small and fast, and since this experiment tests input preservation rather than model intelligence, a tiny model was plenty. Forty repetitions of the filler put the whole prompt at 3464 tokens. Remember that number.

Changing only num_ctx broke recall

I threw the same 3464-token prompt four times, varying only num_ctx across 1024, 2048, 4096, and 8192. The results split cleanly.

num_ctx	prompt_eval_count	secret recall	model answer
1024	1023	failed	"the secret code is None"
2048	2047	failed	"the secret code is ro"
4096	3464	succeeded	`ALPHA-7723-ZULU`
8192	3464	succeeded	`ALPHA-7723-ZULU`

One thing jumped out. With num_ctx at 1024, prompt_eval_count was exactly 1023; at 2048, exactly 2047. My prompt was clearly 3464 tokens, yet the number of tokens the model actually read was clipped to fit num_ctx. Ollama had shaved the over-window input down to the window size. And because the shaved-off side was the head, the secret code hidden at the top disappeared entirely.

No error. No warning. It just said "None," or "ro," with a straight face. Honestly, that's the scariest part. Nothing in the model's answer signals that the input was cut. From num_ctx 4096 up, 3464 fits inside the window, so prompt_eval_count reads a full 3464 and recall is correct. The threshold sat between 2048 and 4096, right on top of my prompt length of 3464.

Why the front, not the back

It felt backwards at first. When something "gets cut," you'd expect the tail to go, yet here the head vanishes. The reason is in how autoregressive inference works. When an LLM produces the next token, what it leans on most directly is the recent tokens just before it. So when the window runs short, the runtime keeps the newest tokens (the tail) and drops the oldest (the head). The same logic applies to multi-turn chat through /api/chat: the Ollama FAQ notes that when context overflows, it quietly drops the oldest messages first.

The trouble is that the things you least want cut all live at the front. System prompt, role instructions, tool definitions, output-format rules. By convention they go right at the top. And the trimming starts precisely there. If an agent that was cruising through a long conversation suddenly loses its persona or breaks its tool-call format, the model didn't get moody. The system prompt may have been pushed out of the window. That was exactly my notes agent.

There's a practical defense in this. Any instruction you truly cannot afford to lose, put it again near the end, just before the question, instead of only at the front. You're placing it where truncation can't reach. Not elegant, but when you can't control num_ctx it worked surprisingly well.

prompt_eval_count snitches on the truncation

The most useful thing I got from this is elsewhere: truncation leaves a trace in the response. The prompt_eval_count that Ollama returns in the /api/generate response is the number of input tokens the model actually prefilled. If that value is smaller than your sent prompt's token count and clings to num_ctx, it was almost certainly truncated.

Why it matters: nobody normally looks at this number. If the answer comes out plausible, you assume the whole input went in. But even if you've stabilized answers with structured outputs, if the model only saw half the input, you get an answer that's schema-clean but factually wrong. Schema validation passes while the facts are off, which is the nastiest kind of bug to debug.

But the default wasn't 4096

If I'd stopped there, I'd have landed on the usual "Ollama's default num_ctx is 4096, so watch out." But with no num_ctx set at all, the 3464-token prompt recalled fine, and prompt_eval_count read a full 3464. Under the 4096-default lore, 3464 obviously passes, so that's consistent so far. So I grew the input.

filler repeats	prompt_eval_count at default num_ctx	note
70	5911	fits whole
100	8431	fits whole
150	12631	fits whole
200	16383	clipped at 16384
250	16383 (recall failed, answer "secret")	head dropped

If the default were 4096, it should have clipped already at 5911. Instead, 12631 tokens went in fine and it hit the ceiling at 16383 (= 16384 − 1). So on my MacBook, the default num_ctx that Ollama 0.30.7 chose was not 4096 but 16384. Checking the Ollama FAQ and community write-ups, recent versions auto-size the default context to available memory. On a 16GB M1, it landed on 16384.

This isn't trivia. It's a portability problem. An agent that ran great on my 32GB desktop, moved to an 8GB cloud instance, gets a smaller default num_ctx, and the same code silently truncates the same input. You get a hard-to-reproduce incident: fine in local testing, quality collapses after deploy. I land firmly on "don't trust the default, always set it." If a default differs from machine to machine, it's effectively a value you can't trust.

So I added one guard to the code

Nothing fancy. Two things. First, I set num_ctx explicitly on every request. Second, once the response comes back, I check whether prompt_eval_count got close to num_ctx, to catch a likely truncation in the logs right away.

def guarded_generate(prompt, num_ctx=8192, model="melavisions/gemma4:latest"):
    body = {"model": model, "prompt": prompt, "stream": False,
            "options": {"num_ctx": num_ctx, "num_predict": 256}}
    req = urllib.request.Request("http://localhost:11434/api/generate",
            data=json.dumps(body).encode(), headers={"Content-Type": "application/json"})
    d = json.load(urllib.request.urlopen(req, timeout=600))

    used = d["prompt_eval_count"]
    # if it filled over 98% of the window, assume it likely truncated and warn
    if used >= num_ctx * 0.98:
        print(f"[warn] prompt_eval_count={used} ~ num_ctx={num_ctx}: "
              f"input may be truncated. Raise num_ctx or shrink the input.")
    return d["response"]

This guard doesn't prevent truncation. It just stops a truncation from slipping past in silence. In my case, this one line answered within five minutes why the summarizer ignored instructions only on long inputs. Input tokens were crossing the default window and the system prompt at the front was getting cut. Raise num_ctx enough and the same input obeyed the instruction again.

If a post-hoc guard that only fires after the response feels off, you can also count tokens before sending. Ollama has no separate tokenizer endpoint, so I measure length up front by hitting /api/generate with num_predict: 0, generating nothing and reading just prompt_eval_count. One cheap prefill tells me whether the input fits the window. In a RAG pipeline with variable input, that pre-measurement lets you branch: bump num_ctx dynamically, or cut the number of context chunks.

What this experiment doesn't tell you

Let me draw the boundaries honestly. First, I measured with one 3.2B quantized model. Truncation itself is runtime-level behavior independent of the model, but "how plausibly the model hallucinates when the head is cut" will differ by model. A larger one might answer "I don't know."

Second, putting the secret at the very front is a deliberate worst case. In real RAG or agents, the important information isn't always in the head. But the system prompt and tool definitions almost always come first, and their loss is the most damaging, which is why I designed it this way.

Third, the default num_ctx landing on 16384 is a product of my 16GB M1 plus Ollama 0.30.7. It varies with version, memory, and how many models are loaded at once. So the lesson I take isn't "the default is 16384" but "the default differs by environment, so don't trust it."

And honestly, one thing I couldn't resolve remains. I ran the same test through the OpenAI-compatible endpoint (/v1/chat/completions) too, where there's no way to pass options.num_ctx per request, and usage.prompt_tokens reported a different number from /api/generate's prompt_eval_count (3464 versus 2384 for the same text). On top of that, on my machine the head survived even on long input and recall worked. I get that the token accounting differs so the two endpoints can't be compared one-to-one, but why the truncation behavior looked different too, I can't cleanly explain. Either way, a reported issue where num_ctx isn't honored on the OpenAI-compatible API and it silently truncates at 4096 does exist, so if you go through /v1, remember you're fully at the mercy of the server default (OLLAMA_CONTEXT_LENGTH).

It's the same thread as tracking load_duration during cold starts. The numbers Ollama quietly tucks into each response, however thinly documented, are the most honest clues to its real behavior. Just as load_duration snitched on cold starts, prompt_eval_count snitches on truncation. If you're running local models seriously, give these numbers a look.

Why a local LLM''s first reply sometimes takes 10 seconds — I measured the cold start (load_duration)

Jangwook Kim — Fri, 26 Jun 2026 06:41:12 +0000

I have been running a local agent on my MacBook for a few days. When I step away to do something else and come back to the same agent, the first reply is noticeably sluggish. The second and third are fine; only that first one drags. While writing yesterday's post decomposing prefill and generation cost, I wrote that I warmed the model before measuring "so model load time (load_duration) would not contaminate the numbers." That line nagged at me. The thing I deliberately threw away was exactly the delay I feel most often in daily use.

So today I measured the cost I excluded yesterday. The time it takes a model to land in memory, the thing we casually call cold start.

load_duration: the line item you usually don't see

Ollama's /api/generate returns a bundle of timestamps on every response. Yesterday I looked at prompt_eval_duration (prefill) and eval_duration (generation). There is one more at the front: load_duration. As the name says, it is the time spent loading the model.

There is a reason you rarely notice it. Call the same model back to back and from the second call on the model is already resident, so load_duration reads near zero. Leave it idle for a while and Ollama evicts the model from memory (five minutes by default), and the next call resurrects the load cost. That eviction is exactly what I was feeling when "stepping away and coming back" felt slow.

I kept the method simple. To isolate load time, the prompt is one line, Reply with the single word: ok, and num_predict is capped at 8 so generation collapses toward zero. To force a cold state, I call ollama stop <model> right before the request. Then the load_duration of the first call is the cold start.

def gen(model, keep_alive="5m"):
    body = json.dumps({
        "model": model, "prompt": "Reply with the single word: ok",
        "stream": False, "keep_alive": keep_alive,
        "options": {"num_predict": 8, "temperature": 0}
    }).encode()
    req = urllib.request.Request(OLLAMA, data=body,
        headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=600) as r:
        d = json.loads(r.read())
    return d["load_duration"] / 1e6  # nanoseconds -> milliseconds

If you want to check it yourself, one curl line does it. Stop the model, call it once, and pull out load_duration.

ollama stop gemma4:12b-it-qat
curl -s http://localhost:11434/api/generate -d '{
  "model": "gemma4:12b-it-qat", "prompt": "ok", "stream": false
}' | python3 -c 'import sys,json; print(json.load(sys.stdin)["load_duration"]/1e9, "s")'

The value comes back in nanoseconds, so divide by 1e9 for seconds. Run that line across a few models and you immediately feel how the table below shifts on your own hardware.

There is one reason to trust this measurement. Ollama returns load_duration separately from prompt_eval_duration and eval_duration, so load time does not bleed into the prefill or generation numbers. The response's total_duration came out close to the sum of those three, which let me isolate the load cleanly. Yesterday I looked at the middle two; today I focus on just the first.

Cold start by model size

I lined up four Gemma 4 models I had pulled, ordered by size. For each I ran ollama stop, then called it cold three times, plus once warm with the model resident. In seconds:

Model	On-disk size	Cold #1	Cold #3	Warm
melavisions/gemma4	2.0 GB	3.33s	1.55s	0.20s
yinw1590/gemma4-e2b	3.1 GB	3.57s	1.79s	0.38s
gemma4:12b-it-qat	7.2 GB	9.00s	2.82s	0.37s
gemma4:e4b	9.6 GB	9.71s	3.86s	0.37s

The broad pattern is what you would expect: bigger model, longer load. The 9.6GB model's first cold start was 9.7 seconds; the same call warm was 0.37 seconds. A 26x gap. In practice, that means if you leave a 7.2GB local chatbot idle past five minutes and speak to it again, you burn several seconds before a single token appears.

What jumps out is the warm column. Whether the model is 2GB or 9.6GB, warm load_duration sat at 0.2 to 0.4 seconds, basically flat. It does not scale with size. The way I read it, this is not actually re-reading weights; it is the keep_alive bookkeeping overhead of Ollama confirming "this model is still up." It is not a real load, so it ignores size. I won't claim to know exactly what work it represents. But for practical purposes, 0.4 seconds warm is "effectively no load cost," and that is the conclusion the measurement supports.

Why "cold" #1 and #3 differ by 2x

Look at the table again and something is off. I stopped the model and re-measured every single time, yet Cold #1 is nearly twice as slow as Cold #3. The 7.2GB model went from 9.00 to 2.82 seconds, the 9.6GB one from 9.71 to 3.86. Both are labeled "cold," but the numbers disagree.

I got stuck here for a while. I first suspected a measurement bug. The answer was the operating system's page cache. ollama stop only evicts the model from the Ollama process's memory; the OS keeps the model file it already read sitting in RAM as page cache. So Cold #2 and #3 re-read the file from RAM, not disk. Drop the disk I/O entirely and it speeds up.

This matters because the thing we casually call "cold start" is really two different things.

Truly cold: right after a reboot, or when memory pressure has flushed the cache. Weights are read from disk for the first time. This is Cold #1.
Cached cold: the model is evicted from Ollama but the file is still in the page cache. This is Cold #3.

If you do not separate these when benchmarking, the second measurement onward quietly picks up the cached value, and you reach the rosy conclusion "cold start is faster than I thought." A real production server reboots, and swapping between several models flushes the page cache. So when setting an SLA or a cold-start budget, base it on Cold #1, the post-reboot worst case, not Cold #3. Had I not known this and measured once, I would have written the 7.2GB cold start as "2.8 seconds" when the real worst case was 9.

The interesting part is that the gap is much smaller on small models. The 2.0GB model's Cold #1 (3.33s) and Cold #3 (1.55s) differ by about 1.8 seconds, while the 9.6GB model's 9.71s and 3.86s differ by almost 6. More bytes to read from disk means the page cache saves you more time. The bigger the model, the steeper the penalty the "first user after reboot" absorbs. If you plan to serve 13B-class or larger locally, treat this cache dependency as a real operational variable.

keep_alive splits the bill

The most direct lever against cold start is keep_alive: how long to hold the model in memory. I put it at two extremes and hit the same 7.2GB model three times each.

Request	keep_alive="0" (unload each time)	keep_alive="10m" (stay warm)
Request #1	7.10s	2.56s
Request #2	2.55s	0.38s
Request #3	2.55s	0.38s

The contrast is sharp. keep_alive="0" unloads the model immediately after serving a request, so every request is cold. Each one eats 2.5 seconds or more of load up front. Check ollama ps between requests and the model is not in memory.

keep_alive="10m" pays the cold value (2.56s) only on the first request, then drops to 0.38 seconds. It shoves the cold start into request one and serves the rest warm. Request #1 of the keep_alive=0 run spiked to 7.1 seconds because the page cache was also empty at that point, a truly cold start. The effect from the previous section shows up here too.

On the command line, the OLLAMA_KEEP_ALIVE environment variable or the API's keep_alive field controls the same thing. Set it to -1 to keep the model resident indefinitely.

So how should I run a local agent?

Measuring turned a few vague operational hunches into something concrete.

First, for chat or agent use, give keep_alive plenty of room. If the model is evicted every time a user speaks, every turn is a cold start. Adding 2.5 seconds per turn on a 7.2GB model wrecks the conversation. As long as memory allows, set a long keep_alive or pin it with -1. This is a setting you can drop straight onto the deployment from my Ollama plus FastAPI production serving guide.

Second, warm the model once at startup. Have your boot script fire a dummy prompt to pay the cold start in advance, so the first real user never eats the 9 seconds. You cannot avoid paying the cold cost on the first request, but that first request does not have to be a real user.

Third, routing across several models is pricier than it looks. Calling a different model per request triggers a reload each time, and if memory is tight they flush each other's page cache down to a true cold (#1 level). If you build a router that rotates four models, compute load cost times switch count up front.

Fourth, benchmark inference speed only after warming. That is precisely why I warmed before measuring yesterday. Measure once while cold and load_duration stacks 9 seconds on top of prefill and generation, so you cannot tell whether the model is slow or the load is. The same principle held in my output reproducibility experiment. Fix every variable except the one you are measuring.

Fifth, memory and responsiveness are a trade. A long keep_alive erases cold starts past the first request, but that model occupies RAM the whole time. Pin a 9.6GB model indefinitely and you shrink what other work can use; load another model and the page cache gets flushed, reviving cold. So I decided which models stay up first, gave a long keep_alive to the one or two I use most, and kept the rest short. Holding every model warm is a luxury only enough memory affords. Hammering in keep_alive=-1 to drive cold start to zero just returns it as a bigger cold on the next model's load.

Limits and what I still don't know

Let me draw the boundary honestly. These numbers come from one MacBook (Apple Silicon, unified memory). A server with a CUDA GPU has an extra step in the load path, copying from disk through system RAM into VRAM, so the absolute values will differ. Don't transplant my numbers onto other hardware. That said, the structural conclusions should survive a hardware change: cold scales with size while warm does not, page cache splits cold into two kinds, and keep_alive decides every cost past the first request.

Also, I did not dig deep enough into Ollama internals to claim exactly what load_duration sums up. It may include initialization like graph construction, not just the file read. What I can observe is the number the API returns and how it responds to model size, page cache, and keep_alive. That range is the scope of today's measurement. The 0.37 seconds that shows up even when warm is something I guessed at, not confirmed.

Finally, page cache behavior depends on how much free RAM you have. On a memory-tight server, even Cold #2 and #3 could get their cache flushed quickly and slow back down toward Cold #1. My measurement leans toward the optimistic case with ample RAM. Next I want to apply artificial memory pressure and see how long the cache holds. Cold start is not a measure-once topic; it is the kind of cost you have to re-measure per environment.

Why a Local LLM Slows Down as the Conversation Grows — I Split Prefill From Generation

Jangwook Kim — Thu, 25 Jun 2026 06:48:26 +0000

When I run a local agent on my MacBook, responses get noticeably sluggish as the conversation drags on. I knew it felt slower, but I had no idea which stage was slowing down or by how much. So I cracked open the timing fields Ollama returns with every response and measured it.

The punchline first. I sent the same 9,700-token prompt twice. The first call took about 55 seconds to produce its first token. The second took 65 milliseconds. Same input, roughly a 396x difference. That single fact explains almost everything about local LLM latency.

The stopwatch Ollama hides in every response

Most people only touch Ollama through a chat UI or ollama run. But if you call /api/generate with stream:false, the response JSON ships with precise timing fields. They are documented in the Ollama API reference.

prompt_eval_count: number of tokens in the input prompt
prompt_eval_duration: time spent processing the prompt (this is prefill)
eval_count: number of tokens generated
eval_duration: time spent generating those tokens (this is generation)
all durations are returned in nanoseconds

The important part is that inference splits into two stages with completely different cost shapes. Prefill reads my entire prompt in one pass and fills the KV cache. Everything up to just before the first token lives here. Generation then emits tokens one at a time, autoregressively. The two behave differently as context grows, and I wanted to see them apart, not blended into one "it's slow" number.

The measurement script is short and uses only the standard library. It varies the context length, asks the same question, and converts the timing fields into tokens per second.

import json, urllib.request

OLLAMA = "http://localhost:11434/api/generate"
MODEL  = "gemma4:e4b"

def call(prompt):
    body = json.dumps({
        "model": MODEL, "prompt": prompt, "stream": False,
        "options": {"num_predict": 64, "seed": 42, "temperature": 0}
    }).encode()
    req = urllib.request.Request(OLLAMA, data=body,
                                headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req, timeout=900) as r:
        d = json.load(r)
    pe_n, pe_d = d["prompt_eval_count"], d["prompt_eval_duration"] / 1e9
    ev_n, ev_d = d["eval_count"], d["eval_duration"] / 1e9
    return {
        "ctx": pe_n,
        "prefill_s": pe_d,
        "prefill_tps": pe_n / pe_d,   # prefill throughput
        "gen_tps": ev_n / ev_d,       # generation throughput
    }

One trap had to be handled first. If you repeat the same prompt, the cache drives prefill to nearly zero (more on that later). To measure cold prefill honestly, every call has to differ from the very first byte. So I prepended a fresh random ID to each prompt and seeded the filler body differently per run. I picked gemma4:e4b to keep iterations light, and I warmed the model up once before measuring so model load time (load_duration) wouldn't leak into the numbers.

A longer context means a later first token

Cold prefill first. I grew the context from about 200 tokens to 9,700 and timed how long it took to reach the first output token.

Context (tokens)	Cold prefill	Prefill tok/s	Generation tok/s	ms per generated token
200	1.0s	197.8	16.81	59.5
644	3.2s	198.4	16.61	60.2
1,244	6.3s	197.6	16.38	61.1
2,476	12.7s	194.6	16.40	61.0
4,852	25.7s	188.6	15.87	63.0
9,716	54.8s	177.3	15.43	64.8

The left chart is nearly a straight line. As context grew 48x (200 to 9,716), prefill grew from 1 second to 55 seconds, about 54x. Roughly proportional. Intuitive enough: more tokens, more to read.

The interesting bit is the prefill rate in tok/s. At short context it ran at 198 tok/s, but near 10k tokens it fell to 177, about 10% slower. So the cost of processing a single token itself rises with context. As I understand it, attention scales quadratically with sequence length, so reading one word at the end of a long document, while re-attending over everything before it, is heavier than reading a word in a short one. That makes prefill climb a touch steeper than linear.

Here is the first practical lesson. The usual culprit behind a slow local agent is not generation speed, it is prefill. Cram five RAG documents in, or replay the last 20 turns wholesale, and tens of seconds evaporate before the model writes a single character. On cloud APIs this cost is billed as tokens that vary with your data format; locally it is billed straight to my wall clock.

If you use a streaming UI, think of this prefill time as exactly how long the user stares at a blank screen or a loading spinner. Once tokens start flowing, they fill in fairly briskly at 16 a second. The problem is everything up to that first character. The longer the context, the longer the user has to sit through a silence that feels like "did it freeze?" What governs the felt responsiveness of a local chatbot is the length of that silence, not the speed tokens stream at. If nothing appears for 55 seconds at 9,700 tokens of context, that is not a tool you can use conversationally.

Generation quietly gets slower too

The right chart looks almost flat, which is exactly why I want to flag it. Generation fell from 16.81 tok/s to 15.43 tok/s, about 8%. Per token, that is 59.5ms creeping up to 64.8ms.

Why? Each new token in the generation stage re-attends over the entire KV cache built so far. A longer context means more to attend over, so the per-token time inches up. Same root cause: attention is sensitive to length.

Honestly, though, that 8% is a side dish next to prefill. Generating 64 tokens at 10k context took about 4 seconds, but the prefill in front of it was 55 seconds. More than 90% of the felt latency happens before the first token. That is why I see "make the prompt short, and make it cacheable" as a far bigger lever than "swap in a faster model for generation."

Why the second call was 396x faster

The cache was the most striking part of this experiment. I sent the same 4,859-token prompt twice in a row.

Call	prompt_eval_count	Prefill time
First (cold)	4,859	25,751ms
Second (warm)	4,859	65ms

prompt_eval_count reported 4,859 both times. The token count was unchanged, yet prefill time dropped about 396x. The model did not skip reading the tokens; it reused a KV cache it had already computed, so there was nothing to recompute.

This is the prefix KV cache in llama.cpp. Ollama runs on top of llama.cpp, so it inherits the behavior. Two requests that share a leading prefix produce a bit-identical KV cache for that span, so the second request skips the shared prefix and processes only from the point where they diverge. With an identical prompt there is no divergence, so prefill effectively vanishes.

One thing that confused me, worth noting. Even on a cache hit, prompt_eval_count still returns the full 4,859. At first I stared at that number and assumed caching wasn't working. The field to watch is not the count but prompt_eval_duration. To confirm the cache is doing its job, look at prefill time, not token count: send the same prompt twice, and if the second prefill drops to a few milliseconds, you got a hit. Miss this and you can misdiagnose "caching is broken" in an environment where it works fine.

Now the random ID I prepended to measure cold prefill makes sense. The cache only reuses the span that is common from the very front. Change the first byte and everything after it must be recomputed. Put a per-request changing value at the head of your prompt and you break the whole cache.

So how should I build the agent?

After this measurement I changed how I assemble prompts for my local agents. The shortlist:

1. Stable content up front, volatile content at the back. System prompt, tool definitions, fixed instructions, anything identical every turn, goes at the very front. The user's new question or a fresh search result, anything that changes, goes after. That alone lets the leading prefill ride the cache from the second turn on, nearly for free.

2. No timestamps or random IDs at the head of the prompt. Pin a line like "Current time: 2026-06-25 15:23:06" to the top of your system prompt and the first line differs every request, busting the cache each time. If you must include it, push it to the end. This one detail can save tens of seconds of prefill.

3. "Fits in the context window" is not "usable." A model can support 32k, but on my laptop even 10k tokens meant 55 seconds to the first token. For interactive use, size your context budget by measured prefill time, not by the supported limit. When I want conversational responsiveness locally, I keep context within a few thousand tokens.

4. If you truly need long context, pay prefill once and reuse it. For RAG where you ask several questions about the same document, keep the document at a fixed position up front and vary only the question at the back. The first question pays the full prefill, but the rest nearly skip it thanks to the cache. The same ordering helps when you wire a local model to an MCP server to build an agent, so you do not re-prefill the system prompt on every tool-call round trip.

The prompt I actually changed

Abstract rules don't land well, so here is how I reordered the prompt for my local tool-calling agent. Before, the order looked like this.

Current time: 2026-06-25 15:23:06    <- changes every request (cache breaker)
Session ID: 9f3a-...                  <- changes every request
[system prompt, 800 tokens]
[tool definitions, 1,200 tokens]
[recent conversation history]
[the user's new question]

The problem is the first two lines. With the time and session ID at the very front, the first byte of the prompt differs on every request. The 800-token system prompt and 1,200-token tool definitions behind them never change a character turn to turn, yet because the cache broke up front, prefill had to run from scratch every time. By the table above, a 2,000-token prefill is about 10 seconds. Ten seconds a turn, thrown away purely on a layout mistake.

After, I did this.

[system prompt, 800 tokens]          <- fixed, at the front
[tool definitions, 1,200 tokens]     <- fixed
[recent conversation history]        <- mostly fixed (only appended to)
Current time: 2026-06-25 15:23:06    <- changing values pushed to the end
Session ID: 9f3a-...
[the user's new question]            <- the part that changes every time, last

With the fixed block moved up front, from the second turn on the 2,000 tokens of system prompt and tool definitions rode the cache wholesale. The conversation history only appends new messages at the tail, so the common prefix stays long. The result: the only thing that needs re-prefilling each turn is the few hundred newly added tokens. Same model, same hardware, but the per-turn latency dropped visibly. Not one line of code got faster. I just changed the order in which I concatenate strings.

Limits of this measurement

Let me draw the boundary honestly. These numbers come from one MacBook, one model (gemma4:e4b), and one runtime (Ollama). The absolute figures (55 seconds, 16 tok/s) change wholesale with GPU, memory, quantization, and runtime. A bigger model, or structured outputs that return typed objects, would add their own variables.

What I trust is the shape, not the absolutes. Prefill grows nearly in proportion to context, generation slows a little, and an identical prefix becomes nearly free through the cache. I expect all three trends to point the same direction in any environment. I have not yet measured how the cache contends under concurrent requests, or how quantization level affects prefill speed. I'm leaving those for the next experiment.

Stop lumping local LLMs into "fast" or "slow." Split them into prefill-to-first-token and per-token generation, and where to fix things becomes obvious. And most of the fixes, it turned out, were not about changing the model. They were about laying out the prompt so it stays cached.

The single most useful line I took from this experiment: when building a local agent, the first thing to check is not the GPU or the model size, it is whether the front of my prompt stays identical every turn. Pin the front and prefill nearly vanishes from the second turn on, on the same hardware. It is free acceleration that costs not one dollar and not one extra GPU. Until I measured it, I had filed this away as a vague "feeling" and let it slide.

Same Article, 1.4x the Tokens in Korean: Measuring the Non-English Token Tax Across 285 of My Posts

Jangwook Kim — Tue, 23 Jun 2026 06:42:22 +0000

I fed one English sentence in. "I refactored the agent loop to cut token usage." Twelve tokens under OpenAI's o200k_base. Then the same meaning in Korean: "토큰 사용량을 줄이려고 에이전트 루프를 다시 짰다." Twenty tokens. Less than half the characters, but 1.7x the tokens.

I wanted to know whether that was a fluke of one sentence or a structural cost baked into my entire blog. I happened to have the perfect test bed. This blog ships every article in four versions: Korean, Japanese, English, Chinese. That means 285 pairs of semantically identical documents across four languages. A dataset where I can cleanly measure how many more tokens the same content costs when only the language changes, with no translation-quality argument in the way.

So I tokenized all 285 articles times four languages with three real tokenizers. Up front: the non-English token tax is real, it's bigger than I expected, and switching models changes the rate.

Why I had to measure this myself

"Korean costs more tokens than English" is folklore in the community. But ask "how much more" and you get scattered answers. Some say 2x, some say 1.2x. Of course they differ. The text measured was different, the tokenizer was different, and whether code blocks were mixed in was different too.

What I needed wasn't a floating number but the cost that lands when my blog runs through the models I actually use. Translation, summarization, embedding, RAG context injection: nearly every step of my automation pipeline is metered in tokens. If Korean costs 1.4x English, that's a number printed straight onto my monthly bill.

I picked three tokenizers as targets. The modern OpenAI line, o200k_base (GPT-4o / GPT-5 generation); the older cl100k_base (GPT-4 / 3.5 generation); and the Claude tokenizer. All three are BPE family, but they learned vocabulary from different data, so they handle non-English differently. The models I reach for daily fall roughly into these three buckets.

How I measured it (frontmatter stripped, full body in)

No fancy tooling. tiktoken runs offline out of the box, and I loaded the Claude tokenizer from Hugging Face's Xenova/claude-tokenizer. For each Markdown file I cut only the YAML frontmatter and tokenized the entire body (code blocks included). What actually goes into an LLM is the whole body, so I deliberately left it unclean.

import tiktoken
from transformers import AutoTokenizer

o200k  = tiktoken.get_encoding('o200k_base')   # GPT-4o / GPT-5 gen
cl100k = tiktoken.get_encoding('cl100k_base')  # GPT-4 / 3.5 gen
claude = AutoTokenizer.from_pretrained('Xenova/claude-tokenizer')

def strip_frontmatter(text):
    if text.startswith('---'):
        end = text.find('\n---', 3)
        if end != -1:
            text = text[end + 4:]
    return text.strip()

# only slugs present in all four languages (285 sets)
for lang in ['ko', 'ja', 'en', 'zh']:
    body = strip_frontmatter(open(path(lang, slug)).read())
    o200k_tokens  = len(o200k.encode(body))
    cl100k_tokens = len(cl100k.encode(body))
    claude_tokens = len(claude.encode(body))

I only counted the 285 articles that have the same filename in all four languages. That removes any sampling imbalance from articles that exist in English but not Korean. It's a strict 1:1:1:1 comparison of the same article's four versions.

What the numbers say

Totals across all 285 articles. Unit is tokens.

Language	o200k (modern)	cl100k (older)	Claude	Characters
English (en)	908,938	915,128	1,003,948	3,859,685
Chinese (zh)	1,045,977	1,267,007	1,340,943	2,493,687
Japanese (ja)	1,217,284	1,502,403	1,579,075	2,584,255
Korean (ko)	1,256,718	1,668,007	1,882,800	3,076,489

Set English to 1.0 and the tax snaps into focus.

Language	o200k ratio	cl100k ratio	Claude ratio
English	1.00	1.00	1.00
Chinese	1.15	1.39	1.34
Japanese	1.34	1.64	1.57
Korean	1.38	1.82	1.88

Even on the modern tokenizer (o200k), Korean is 1.38x English and Japanese 1.34x. On the older one (cl100k), Korean stretches to 1.82x. On the Claude tokenizer, Korean is the most expensive of the three at 1.88x.

The part I found striking is that character count and token count pull apart. The English versions are the longest by characters, at 3.86M. Korean is shorter at 3.08M. Yet Korean uses 38% more tokens. Fewer characters, more tokens. By tokens-per-character: English 0.235, Korean 0.408, Chinese 0.419, Japanese 0.471. English BPE crams common words into a single token each, but Hangul, kana, and Han characters weren't learned that way, so they shatter into more tokens.

Why Korean eats more tokens: I cut "에이전트" apart

I wanted to see the cause with my own eyes, so I fed one word to the tokenizer and decoded how it splits.

w = "에이전트"   # = agent
[o200k.decode([t]) for t in o200k.encode(w)]
# ['에', '이', '전', '트']   -> 4 tokens

English "agent" is one token. Korean "에이전트" splits into four tokens, one per syllable block. cl100k gave the same result. In English a word is roughly a token; in Korean a character (syllable block) is closer to a token. That 4:1 gap on a single word, accumulated across a whole article, becomes 1.4x.

One short sentence across all three tokenizers makes it even clearer. Every version means "I refactored the agent loop to cut token usage."

Language	Chars	o200k	cl100k	Claude
English	47	12	12	12
Chinese	19	14	24	21
Japanese	30	22	28	27
Korean	28	20	31	32

English is 12 across all three. Once you go non-English, the tokenizers diverge. The same Korean sentence is 20 on o200k and 32 on Claude. A 1.6x swing decided by one model choice.

Japanese has its own texture. By tokens-per-character it's the highest of the three at 0.471. Han characters, hiragana, and katakana mix in a single sentence, and katakana loanwords ("エージェント") split syllable by syllable. Yet by full-document total, Korean uses more tokens than Japanese. Japanese packs heavy meaning into single Han characters so documents come out shorter, while Korean documents are simply longer, so even with better per-character efficiency than Japanese it overtakes on total volume. Efficiency and total don't point the same way.

OpenAI's tokenizer jump was really a non-English discount

This was the most unexpected find of the experiment. I computed how much each language's token count dropped when the tokenizer moved from cl100k to o200k.

English: down 0.7%
Chinese: down 17.4%
Japanese: down 19.0%
Korean: down 24.7%

For English users, swapping tokenizers barely moves the count. 0.7% is noise. But Korean got a quarter cheaper on the same articles. What OpenAI actually did by growing the vocabulary for o200k was quietly hand non-English a discount. English was already near-optimal, so there was nothing left to squeeze.

This feeds a real decision. For a workload that runs a lot of CJK text, when you pick a model don't just read benchmark scores; check the tokenizer generation too. Same price tag, but on non-English text the actual bill is decided by the tokenizer. My earlier measurement of how data formats move token cost taught the same lesson. A model's price tag is only the start; the real cost is set by how the input turns into tokens.

The tax varies per article, so don't trust the average

The totals are tidy, but split by article the variance is large. For each of the 285 articles I computed the Korean-to-English token ratio and took median and mean separately (o200k).

Language	Per-article median	Per-article mean
Chinese	1.10	1.14
Japanese	1.41	1.40
Korean	1.31	1.40

Korean has a median of 1.31 but a mean of 1.40. Mean above median means a handful of token-heavy articles drag the average up. Looking closer, those were articles dense in Korean prose with little English code or proper nouns. Tutorial articles full of code blocks sat near 1.1, because code is English anyway and tokenizes almost identically across all four versions.

A practical lesson: estimate one article's cost from the corpus-average ratio and you'll miss badly on prose-heavy, code-light pieces. For a single job where cost matters, tokenize that specific text.

So what shows up on my bill

This blog starts a new post from an English draft, renders it into three other languages, and embeds the whole corpus for related-post recommendations and search. Almost every step is token-metered.

Across the corpus, the 285 English versions are about 0.91M tokens on o200k. All four languages together are about 4.43M tokens. Versus running English-only, multilingual isn't simply 4x; the non-English tax stacks on top. Look at just the three translations (ko+ja+zh) and it's about 3.52M tokens, where estimating from three English copies (2.73M) would have undercounted by about 29%.

Here's the mistake I kept making: eyeballing Korean cost from English token counts. That's off by 28% even on a modern model, and nearly half on an older one. Quote in English, get billed in Korean, and the gap compounds every month.

Concretely: picture translating one new post from an English draft into Korean. The input is the English source (about 3,200 tokens) and the output is the Korean translation. The Korean output carries 1.38x the tokens for the same content. Output tokens usually cost more than input, so the non-English tax lands on the most expensive side. Push that into Japanese and Chinese and the tax applies three separate times. In my one-article-four-languages structure, non-English output tokens are a much bigger chunk of total publishing cost than English's share. When I added Chinese and the article count quadrupled, the reason cost jumped by more than exactly 4x lives right here.

The fixes I'm making to shrink the tax. My recommendation pipeline resends the same context repeatedly, so I turned on prompt caching to keep the non-English tax from re-applying on every call. RAG chunks get cut by real token count, not character count. Invert the tokens-per-character figure and you get chunk size: with a 512-token embedding context, English fits about 2,170 characters but Korean hits the same limit at about 1,250. Cut Korean at the same "1,000 characters" as English and the chunk holds 1.7x the tokens, silently overrunning the context window or getting truncated. That's exactly why I revisited chunk sizing in my Korean RAG embedding writeup.

Where this measurement doesn't reach

Honest limits. First, token count is only one axis of cost. The same token has a different price per model, and input and output prices differ too. This post measured "how many tokens" only; "so how many dollars" you have to plug into your own model and plan.

Second, more tokens isn't automatically a loss. CJK fits the same information into fewer characters. The proof is that the Korean versions have fewer characters than English yet carry the same meaning. Token efficiency and information density are separate stories.

Third, my corpus is a tech blog, so the bodies are heavy with English code, proper nouns, and technical terms. Pure everyday Korean prose could show a bigger multiplier. So this 1.38x is "my environment, technical-document baseline," not a universal constant. If you want to dig deeper into tokenizer behavior, my post on BPE quantization and merging is an adjacent starting point.

Fourth, models keep changing. If the next tokenizer generation grows its CJK vocabulary, this gap narrows again. So the real conclusion here isn't "Korean is 1.38x." It's "don't estimate; run your own text through your own model's tokenizer." The code is all above. Wiring it to your own corpus takes about ten minutes.

Why this token-level sense of measuring and reproducing output matters runs through my experiment on output reproducibility with temperature and seed in the same spirit. Working with an LLM is, in the end, counting tokens. Knowing the counting unit differs by language before you start, versus finding out later, is the difference your monthly bill remembers.

Does a Local LLM Ever Repeat Itself? Measuring Output Reproducibility with Temperature and Seed

Jangwook Kim — Mon, 22 Jun 2026 06:57:17 +0000

I ran my evaluation script twice and got two different scores. Same code, same prompt, same model. Nothing changed, yet one previously passing case failed.

My first guess was that I had touched something. But a re-run passed again. That moved my suspicion to the model itself. An LLM is not a function that maps the same input to the same answer. I knew this intellectually, but once my eval pipeline started wobbling, the question got concrete fast: what exactly do I need to pin down to make the output reproduce?

So I measured it myself. Not on a cloud API, but in a local Ollama + Gemma 4 setup I could control end to end. I sent the same prompt dozens of times and bucketed the outputs by hash to count how many distinct ones appeared. Up front: in my environment, reproducibility came down to exactly two knobs.

Why results wobble when you changed nothing

The final step where an LLM picks a token is sampling, drawing one option from a probability distribution. The knob that changes the character of that sampling is temperature. At temperature 0 the model just takes the highest-probability token every time (greedy). There is no room for randomness, so in theory it should be deterministic. Raising temperature opens room for the second- and third-ranked tokens, and the thing that governs that lottery is seed.

There is one more knob worth naming: num_predict, the maximum number of tokens to generate. When it is short, there are fewer points where the model can diverge, so it looks more deterministic; when it is long, tiny differences have more room to accumulate toward the tail. So I grabbed a clean signal first with a short tagline (about 40 tokens) and tested long outputs separately. That long-output test is where I hit the empty-response problem I will get to later.

All of that is straight from the docs. The question is whether the theory actually holds on my laptop. Just browsing the ollama issue tracker, you find a steady stream of reports: "I fixed the seed but the answer changes" (#4660), "temperature=0 with a fixed seed still differs between the first and second run" (#586). So I decided not to trust it and to measure instead.

The experiment: the same prompt, dozens of times

I built the test environment in a temporary directory outside the repo. Ollama 0.30.7, Apple Silicon, two models: a small 2GB Gemma 4 build and the 9.6GB gemma4:e4b. The point of using two sizes was to see whether the same pattern shows up regardless of model scale.

The method is simple. Send the same prompt 12 to 15 times per condition, hash each output with SHA-256, and count how many distinct hashes appear. One means fully deterministic; a larger number means the output is scattering.

import json, hashlib, urllib.request
from collections import Counter

def gen(model, prompt, temperature, seed=None, num_predict=40):
    opts = {"temperature": temperature, "num_predict": num_predict}
    if seed is not None:
        opts["seed"] = seed
    body = {"model": model, "messages": [{"role": "user", "content": prompt}],
            "stream": False, "options": opts}
    req = urllib.request.Request("http://localhost:11434/api/chat",
            data=json.dumps(body).encode(),
            headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as r:
        return json.load(r)["message"]["content"].strip()

def count_unique(model, prompt, temperature, seed, n):
    outs = [gen(model, prompt, temperature, seed) for _ in range(n)]
    hashes = [hashlib.sha256(o.encode()).hexdigest()[:12] for o in outs]
    return len(set(hashes)), Counter(hashes).most_common(1)[0][1]

I split this into five conditions. For temperature=0 and temperature>0, a fixed seed and no seed, plus a final case where I keep the seed fixed but spin up a fresh Python process and run once more. That last condition matters, because "reproducing inside the same process" and "reproducing after you kill and restart the process" are completely different guarantees from an eval and CI standpoint.

I picked a generative prompt with room to vary: "Write a single short marketing tagline for a new AI coding assistant." Diversity only shows up on a task that does not collapse to a single right answer. Had I used a closed question like "what is 2+2," raising temperature would barely scatter anything, and the seed's effect would be invisible.

I tracked two metrics together. One is the distinct count above; the other is majority share, the fraction of all N runs taken by the single most frequent output. A distinct count of 5 with one output appearing 11 times is effectively stable; five outputs scattered evenly drops majority share toward 0.3. To compress the shape of the distribution into one number, watching both felt safer.

Results: there were only two knobs

The table makes the pattern sharper.

Condition	gemma4 ~2GB (N=15)	gemma4:e4b 9.6GB (N=12)
temperature=0, no seed	1 (deterministic)	1 (deterministic)
temperature=0, fixed seed	1	1
temperature>0, no seed	5	7
temperature>0, fixed seed	1	1
temperature>0, fixed seed (process rerun)	1	1

Here is the story it tells. At temperature 0 both models produced exactly one kind of output. Fifteen runs, twelve runs, not a character off, the same sentence every time. Seed or no seed, the result was identical. That confirms directly that the seed has nothing to do under greedy decoding.

The only cell that scattered was temperature>0 with no seed. The 2GB model split into 5 distinct outputs out of 15, the 9.6GB model into 7 out of 12. But fix the seed to 42 at the same temperature and it snapped back to 1. The most striking part is the last row. I spun up a brand-new Python process and ran it again, and with the same seed it produced the same sentence: "Code faster, effortlessly smart." Two independent executions matched character for character.

The actual sentences make it tangible. At temperature 0 the 2GB model emitted only "Code Smarter, Not Harder" every time. Raise temperature to 0.8 and drop the seed and variations like "Code Smarter, Not Harder with Ada" slipped in. Pin the seed to 42 and it froze on that one line, which held across the process restart. The 9.6GB model behaved the same, splitting into 7 variants at seed-less temperature 0.8 and converging to "Code faster, effortlessly smart" once the seed was fixed. The contrast is even clearer in majority share: the 9.6GB model at seed-less temperature 0.8 sat at 0.333, meaning even its most common output showed up only one run in three. Every other condition was 1.0, all identical.

The result was clean enough that I doubted it again, so I swapped models and reran. Two models that differ 5x in size showed the same pattern. In my environment at least, the two knobs of temperature and seed were enough to control reproducibility.

One empty response taught the bigger lesson

I had originally meant to lean on the 12B gemma4:12b-it-qat as the main model. But that community build returned nothing but an empty string. On both /api/generate and /api/chat, done_reason came back length and eval_count climbed to 40 or 200, yet the content was an empty string.

content repr: ''
done_reason: length   eval_count: 200

Tokens were clearly generated. The GPU spun for 28 seconds each time. But nothing reached the user-visible text. The chat template on this QAT build is probably broken, or the model emitted only invisible control tokens. The exact cause is outside my expertise, so I will not assert one.

The lesson, though, is clear. "The model ran" and "the model answered" are different statements. A pipeline that treats a rising eval_count as success would have let this empty response slip straight into the evaluation data. Why you need one guard that rejects zero-length responses, this failure argued more convincingly than any line of code. Failure is content too, and I felt it again here. I have hit this kind of packaging variable before with local models, in a similar shape as the small-model schema limits I wrote about in the post on Ollama structured outputs.

Right locally does not mean right on the cloud

Let me draw the line clearly. What I measured is determinism "on local Ollama, sending requests one at a time, sequentially." Move that condition onto a cloud API like OpenAI or Claude and the story changes.

The most convincing explanation of why comes from Thinking Machines' "Defeating Nondeterminism in LLM Inference" (September 2025). People often say "GPU floating-point math is non-deterministic, that's why," but the piece locates the real cause elsewhere. Inference servers batch many users' requests together, and the batch size shifts unpredictably with server load at that moment. Because the core kernels take a slightly different numerical path depending on batch size, the same prompt can diverge into different tokens even under greedy decoding. As I understand it, the real culprit of non-determinism is not randomness but the fact that your request lands in a differently sized batch each time.

Local ollama is not a perfectly safe zone either. Issue #586 reports that with the same seed, same temperature=0, and same num_ctx, the output still differed slightly between the first and second run, and more interestingly that the same code produced a different "fixed" output on Ubuntu versus Windows. In other words, determinism may be a platform-bound property. My measurements were probably clean because they ran on one Mac, with one version of ollama, on short outputs. The longer the output and the larger num_ctx, the more room tiny numerical differences have to accumulate and diverge.

I did not reproduce this batch non-determinism directly. I never built an environment that applies concurrent load. So I cite that part from the writeup only. The range I verified by hand is strictly "sequential requests, local, short outputs." Blur that boundary and my article becomes one more piece selling unverified claims as fact.

So honestly, building eval reproducibility on top of a cloud API is trickier than it sounds. Even an API that accepts a seed parameter does not protect you from batch non-determinism. That, I think, is the root reason LLM evaluation never settles as cleanly as a unit test.

What I applied to evaluation and agent testing right away

Here is what I folded into my workflow as soon as the measurement was done.

First, pin regression evals at temperature=0 with a fixed seed. For a regression test that checks "did it change" rather than "how good is it," you do not need the model's creativity. Better to lock onto one reproducible output and catch the moment it shifts. In my environment this combination returned the same sentence even after a process restart, which is enough to put in CI.

Second, never conclude from a single run. Features that run at a higher temperature, where diversity is the value, scatter by nature. To evaluate that output you cannot run it once and call it pass or fail; you run it N times and look at the distribution. Seeing it split 7 out of 12 in my measurement tells you how risky it is to judge a feature on one lucky output.

Third, put an output-validity guard in front of the eval. The empty-response 12B model is the direct reason. Unless you explicitly classify zero length, JSON parse failure, or a missing expected field as a failure, a broken model masquerades as a healthy score. The skeleton I use for regression tests is about this simple.

def assert_reproducible(model, prompt, expected_hash, n=5):
    outs = [gen(model, prompt, temperature=0, seed=42) for _ in range(n)]
    # 1) empty-response guard: separate "ran" from "answered"
    assert all(len(o) > 0 for o in outs), "empty output detected"
    # 2) does it lock to one kind within a single run?
    hashes = {hashlib.sha256(o.encode()).hexdigest()[:12] for o in outs}
    assert len(hashes) == 1, f"non-deterministic: {len(hashes)} variants"
    # 3) does it match the baseline I pinned earlier?
    assert hashes.pop() == expected_hash, "output drifted from baseline"

Pin the expected hash once and CI catches the moment a model version or an ollama upgrade changes the output. That is my rebuttal to the common resignation that "you can't test an LLM." You can't test all of it, but the part where you control the reproducibility conditions, you certainly can.

It extends to agents the same way. To regression-test an agent's sequence of tool calls, that sequence has to reproduce. If you have built a fully offline MCP server on a local model, fixing the seed on top of it to reproduce tool calls is relatively controllable. An agent on a cloud LLM, by contrast, struggles to get the same guarantee because of batch non-determinism. In the end, "where you run inference" decides "how tightly you can write the test," and that was the most practical takeaway from this experiment.

Next I plan to build an environment that applies concurrent load and check directly whether batch non-determinism reproduces even on local ollama. If it does, my tentative conclusion that "local is safe" will earn a footnote.

Stop Feeding Raw JSON to Your LLM — I Measured Token Cost Across 9 Data Formats

Jangwook Kim — Sun, 21 Jun 2026 06:34:53 +0000

I needed to hand an agent a 50-row product catalog as context. Out of habit I dumped it with json.dumps(records, indent=2), and the token counter read past 4,000. The data itself was tiny. I started to suspect the indentation and quotes were eating close to half my tokens. So I serialized the exact same data into nine formats and counted the real tokens.

Up front: for flat data, TSV is 62% cheaper than pretty JSON. But the moment the data nests, that conclusion reverses entirely. I went looking for where that boundary actually sits.

The setup: count with tiktoken, don't guess

Token cost gets tossed around as "roughly chars × 0.75," but that heuristic completely misses per-format differences. So I used OpenAI's own tiktoken. I ran two encodings side by side: o200k_base (GPT-4o, the o-series, the GPT-5 family) and the older cl100k_base (GPT-4 and 3.5).

The test data mimics a realistic "tool result." Fifty product records, each a flat object with nine fields: id, sku, name, category, price, stock, warehouse, status, rating.

import json, io, csv, yaml, tomli_w, tiktoken

records = json.load(open("records.json"))  # 50 flat records
enc = tiktoken.get_encoding("o200k_base")  # GPT-4o / GPT-5 family

def tok(s): return len(enc.encode(s))

print("pretty :", tok(json.dumps(records, indent=2)))
print("compact:", tok(json.dumps(records, separators=(",",":"))))
print("csv    :", tok(to_csv(records)))

I ran the sandbox in a throwaway mktemp -d directory outside the repo, kept only the result logs and the chart, then wiped the environment. The habit of isolating one-off experiments like this hardened during the stretch where I ran eight agents and tracked their real cost. Cutting tokens is, in the end, one line in that same ledger.

Flat data: TSV wins by a mile

Here are the measured results for serializing 50 flat records nine ways. On o200k_base, pretty JSON (4,128 tokens) is the 0% baseline.

Format	Chars	o200k tokens	cl100k tokens	vs pretty JSON
TSV	3,742	1,568	1,663	−62.0%
CSV	3,742	1,650	1,650	−60.0%
Markdown table	4,766	1,897	1,897	−54.0%
JSON (compact)	7,985	2,578	2,593	−37.5%
key: value lines	6,982	2,708	2,708	−34.4%
YAML	7,834	3,159	3,159	−23.5%
TOML	8,533	3,176	3,191	−23.1%
JSON (pretty)	10,986	4,128	4,143	0.0%
XML	13,654	4,777	4,778	+15.7%

The thing that jumps out is the 2.6x gap between pretty JSON and TSV. Same information. Not one fact the model receives changes. And yet two-space indents, repeated key names, quotes, and braces inflate the token count 2.6 times. XML goes further still, writing every field twice thanks to closing tags, landing 16% above even pretty JSON. I think using XML as an LLM input format is a near-guaranteed loss.

Why CSV, TSV, and Markdown tables are cheap is simple. They write the field names once in a header row, then list only values across the 50 rows. JSON-family formats repeat a key like "warehouse": 50 times, once per record. The more fields and the more rows, the worse that repetition tax gets.

Break it down per record and it's starker. Pretty JSON's 4,128 tokens spread across 50 records is about 82 tokens each. Strip TSV's header row (~18 tokens) and 1,550 tokens over 50 rows is about 31 tokens per record. The same nine fields on one line, and one side spends 82 tokens while the other spends 31. The difference isn't the data. It's nine key names repeated 50 times, plus the quotes and braces wrapping them. To the model, "category":"books" and books are the same fact, but the former spends three or four times the tokens to convey it.

It's not the repeated keys that cost, it's the boilerplate

I want to correct a likely misread here. The accurate framing isn't "JSON is slow," it's "structural punctuation is expensive." Compact JSON beats pretty JSON by 37.5% not because it has fewer keys. The keys still repeat 50 times. What disappears is the indentation whitespace and the line breaks.

2-space indent "  "  -> 1 token (id 220)
3 commas       ",,,"  -> 1 token

On o200k_base, a two-space indent is itself a single token (id 220). When 50 records each indent nine fields, that whitespace token alone gets laid down hundreds of times. Add a newline per line and an opening and closing brace per object. To a human that's readability; to the model it's pure cost. So I've made "no pretty-printing unless a human is going to read it" my default.

With nested data, the conclusion flips

If I'd stopped here I'd have walked away with the wrong lesson, "always CSV." So I measured a differently shaped dataset. Twenty orders, where each order holds a customer object (with a nested address) and a variable-length array of line items.

Format	o200k tokens	vs pretty JSON
JSON (compact)	1,538	−45.7%
YAML	1,958	−30.9%
TOML	2,021	−28.7%
JSON (pretty)	2,835	0.0%

CSV, TSV, and Markdown tables drop out of the running entirely. There's no way to cram variable-length item arrays and nested objects into a two-dimensional grid. And compact JSON, which sat mid-pack at −34% on flat data, takes first place at −45.7% on nested data. YAML, flat or nested, was never as cheap as I'd assumed, thanks to its indentation cost. The folk wisdom that "YAML is both human-readable and token-light" did not hold up, at least not in this measurement.

So the boundary is this. Uniform rows, use a tabular format (CSV/TSV/Markdown); nested structure, use compact JSON. That one line is the most useful rule I pulled out of today's experiment.

In an agent loop, this compounds

A few thousand tokens sounds like nothing, but an agent resends the same context every turn. Say you pin a 50-item catalog into the system prompt as pretty JSON and run a 30-turn conversation. Switching the format to TSV alone drops about 2,560 tokens per turn (4,128 → 1,568). Over 30 turns that's 76,000 tokens. On a model with a tight context window, that's the difference between fitting and not; on a metered model, it's input-token cost, dollar for dollar.

Here's a likely objection: "Doesn't prompt caching make the same context cheap anyway?" It does. If the catalog is pinned in the system prompt, a cache hit cuts the cost sharply. But caching doesn't reduce the token count itself. Cached tokens still occupy the full context window, and caches usually expire after a short TTL and then refill at full price. On top of that, tool results that change every turn can't be cached in the first place. Trimming tokens via format isn't a technique that competes with caching, it's one that complements it. Turn both on and you win twice.

This matters most when an MCP tool returns a large result. When a server you built with FastMCP returns a DB query as raw JSON, that format is the model's input cost. One small decision on the server side, serializing a flat result as CSV or TSV, shifts the token ledger of the whole agent.

The limits are real, of course. I only measured token counts; I did not measure whether the model understands each format equally well. That's as far as I could reproduce without API calls. My intuition is that a format like CSV, where the header sits far from the values, could confuse the model on field-heavy data. With the header only once at the top, the model has to count by position to know what the 7th value in the 30th row means. Save tokens and lose accuracy and you haven't gained anything. So before applying this for real, run token savings and response quality together as an A/B at least once. I'm deferring that check to the next experiment.

What I'm applying starting tomorrow

Today's measurements changed my defaults to this.

Flat record arrays going into context: CSV or a Markdown table. No pretty JSON.
Nested structures: json.dumps(x, separators=(",",":")), compact. indent=2 only when a human is debugging.
No XML as LLM input. It spends the most tokens on the same information.
If a format change cut tokens significantly, verify once that model accuracy holds in that format.

Data format is a value your code picks almost automatically, so you rarely think about it. But the moment it enters an LLM context, that thoughtless indent=2 can become half your token bill. Until I measured it myself, I was underestimating the size of it too.

The measurement code and full logs were run once in the sandbox and preserved in docs/evidence/llm-token-cost-data-format-experiment.md. Measured on tiktoken 0.12.0, Python 3.12.8.

Building a TypeScript MCP Client from Scratch — @modelcontextprotocol/sdk v1.29 in Practice

Jangwook Kim — Fri, 19 Jun 2026 06:38:30 +0000

I've always been curious what Claude Desktop does internally when it connects to an MCP server. "It connects via stdio" is the short answer, but I needed to see the code to actually understand it. So today I installed @modelcontextprotocol/sdk, built a TypeScript MCP client from scratch, and ran it against a server I wrote myself.

The verdict: it's a lot simpler than I expected. There was also one behavior that surprised me — error handling works differently from what I assumed.

The idea: do what Claude Desktop does, manually

MCP (Model Context Protocol) is the standard interface for AI agents to access external tools and data. I've written a lot about building MCP servers — including how to build one in TypeScript and spinning one up with Python FastMCP in 30 minutes. But I've never written about implementing the client side myself.

Thinking about production use cases, there are clear situations where a custom MCP client makes sense:

Calling MCP server tools automatically from a CI/CD pipeline
Integrating an MCP server into a custom agent layer you're building yourself
Using MCP server capabilities as a library inside existing Python or TypeScript code

You don't need Claude Desktop or Claude Code. The @modelcontextprotocol/sdk package contains everything needed to build a client.

Installing the SDK — two classes are all you need

npm install @modelcontextprotocol/sdk zod

The installed version is 1.29.0 as of today. The SDK includes both server and client implementations.

There are two core classes for building an MCP client.

Client — Manages the logical connection to a server. Provides methods like listTools(), callTool(), listResources(), and readResource().

StdioClientTransport — The transport layer for communicating with stdio-based MCP servers. It spawns the server process directly using command and args.

To connect to remote MCP servers (HTTP/SSE), you'd use a different Transport class. This guide covers stdio only.

Setting up the demo — server and client, both from scratch

I built a simple MCP server for the demo. It has two tools: calculate (basic arithmetic) and transform_text (string transformations).

// server.mjs
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
import { z } from "zod";

const server = new McpServer({ name: "demo-tools", version: "1.0.0" });

server.tool(
  "calculate",
  "Basic arithmetic: add, subtract, multiply, divide",
  {
    operation: z.enum(["add", "subtract", "multiply", "divide"]),
    a: z.number(),
    b: z.number(),
  },
  async ({ operation, a, b }) => {
    const ops = {
      add: a + b,
      subtract: a - b,
      multiply: a * b,
      divide: b === 0 ? null : a / b,
    };
    const result = ops[operation];
    return {
      content: [{
        type: "text",
        text: result === null
          ? "Error: division by zero"
          : `${a} ${operation} ${b} = ${result}`,
      }],
    };
  }
);

server.tool(
  "transform_text",
  "Text transformation: uppercase, lowercase, reverse, word_count",
  {
    text: z.string(),
    op: z.enum(["uppercase", "lowercase", "reverse", "word_count"]),
  },
  async ({ text, op }) => {
    const results = {
      uppercase: text.toUpperCase(),
      lowercase: text.toLowerCase(),
      reverse: [...text].reverse().join(""),
      word_count: `Word count: ${text.trim().split(/\s+/).length}`,
    };
    return { content: [{ type: "text", text: results[op] }] };
  }
);

server.resource(
  "server-info",
  "mcp://demo/info",
  async (uri) => ({
    contents: [{
      uri: uri.href,
      mimeType: "text/plain",
      text: "MCP demo server v1.0.0 — stdio transport",
    }],
  })
);

const transport = new StdioServerTransport();
await server.connect(transport);

There's no need to run the server separately. The client's StdioClientTransport spawns the server process automatically.

Client implementation — listTools, callTool, listResources

// client.mjs
import { Client } from "@modelcontextprotocol/sdk/client/index.js";
import { StdioClientTransport } from "@modelcontextprotocol/sdk/client/stdio.js";

const transport = new StdioClientTransport({
  command: "node",
  args: ["server.mjs"],
  cwd: process.cwd(),
});

const client = new Client(
  { name: "demo-client", version: "1.0.0" },
  { capabilities: {} }
);

await client.connect(transport);

The moment client.connect(transport) is called, node server.mjs is spawned as a subprocess. The client and server then exchange JSON-RPC 2.0 messages over stdin/stdout pipes.

Once connected, I ran three operations in sequence.

1. List available tools

const { tools } = await client.listTools();
for (const t of tools) {
  const params = Object.keys(t.inputSchema?.properties ?? {}).join(", ");
  console.log(`  • ${t.name}(${params}) — ${t.description}`);
}

2. Call a tool

const result = await client.callTool({
  name: "calculate",
  arguments: { operation: "multiply", a: 42, b: 7 },
});
console.log(result.content[0].text);

3. List and read resources

const { resources } = await client.listResources();
const info = await client.readResource({ uri: "mcp://demo/info" });
console.log(info.contents[0].text);

Here's the actual output from the sandbox run:

=== MCP Client Demo — @modelcontextprotocol/sdk v1.29.0 ===

✓ Connected to MCP server

Found 2 tool(s):
  • calculate(operation, a, b) — Basic arithmetic: add, subtract, multiply, divide
  • transform_text(text, op) — Text transformation: uppercase, lowercase, reverse, word_count

--- calculate tool ---
  42 multiply 7 = 294
  100 divide 4 = 25
  999 add 1 = 1000

--- transform_text tool ---
  "Model Context Protocol" → MODEL CONTEXT PROTOCOL
  "BUILD ONCE RUN EVERYWHERE" → build once run everywhere
  "hello world from MCP" → Word count: 4

Found 1 resource(s): mcp://demo/info
  Content: MCP demo server v1.0.0 — stdio transport

✓ Client closed cleanly.

The server was never started separately. The client spawned node server.mjs, communicated with it, and terminated both processes cleanly when client.close() was called.

Errors come back as isError, not as exceptions

This is the behavior that surprised me. When you call a tool that doesn't exist, callTool() doesn't throw — it returns a response object with isError: true.

const result = await client.callTool({
  name: "nonexistent_tool",
  arguments: {},
});
console.log(result);
// {
//   content: [{ type: "text", text: "MCP error -32602: Tool nonexistent_tool not found" }],
//   isError: true
// }

No need for try/catch around tool calls. Instead, check result.isError in every tool call handler.

async function callToolSafe(client, name, args) {
  const result = await client.callTool({ name, arguments: args });
  if (result.isError) {
    throw new Error(result.content[0]?.text ?? "Unknown MCP error");
  }
  return result.content[0]?.text;
}

This is intentional per the MCP spec. Tool execution errors and protocol errors are kept separate: protocol-level errors might throw, but tool-level errors come back in the content. Once I understood this, it made sense — it matches how Claude agents receive tool output. Even when a tool fails, the LLM gets the error text as part of the context and can reason about it.

Parallel callTool — 4 calls in 1ms

I also tested parallel calls with Promise.all.

const start = Date.now();
const ops = [
  ["add", 1, 1],
  ["multiply", 12, 12],
  ["subtract", 100, 37],
  ["divide", 144, 12],
];
const results = await Promise.all(
  ops.map(([op, a, b]) =>
    client.callTool({ name: "calculate", arguments: { operation: op, a, b } })
  )
);
console.log(`${ops.length} parallel calls completed in ${Date.now() - start}ms`);

Output:

Parallel calls (4 ops) in 1ms:
  1 add 1 = 2
  12 multiply 12 = 144
  100 subtract 37 = 63
  144 divide 12 = 12

Even over stdio, the SDK handles request multiplexing internally. Four concurrent calls go out and get matched to their responses correctly. Keep in mind that with stdio the server still processes them one at a time — if your tools are CPU-heavy, parallel calling has limited upside.

Three real-world situations where a custom client is useful

Implementing this clarified where a custom client actually makes sense.

Automation scripts calling MCP tools

If your MCP server exposes code linting, file conversion, or external API lookups, you can invoke those tools from GitHub Actions or a local shell script — just a small Node.js program running node client.mjs. No GUI required.

Building a custom agent framework

If you're writing your own agent loop without LangGraph or LlamaIndex, a custom MCP client slots in as the tool execution layer. Pull the tool list with listTools(), inject it into your LLM prompt, parse the model's tool call decision, and run it with callTool(). The MCP Gateway post is a natural follow-up if you need to route traffic across multiple servers.

Testing and debugging MCP servers

During MCP server development, a custom client lets you verify tool behavior quickly without Claude Desktop. Call listTools() to check the generated inputSchema, then fire callTool() with various parameter combinations.

Connecting to an existing public MCP server

So far I've only connected to my own server. The same client works with any public MCP server package.

npm install @modelcontextprotocol/server-filesystem

Then update the transport:

const transport = new StdioClientTransport({
  command: "npx",
  args: [
    "-y",
    "@modelcontextprotocol/server-filesystem",
    "/path/to/allowed/directory",
  ],
});

With this you can call read_file, write_file, and list_directory tools via callTool(). The command/args in Claude Desktop's config file maps directly to what StdioClientTransport accepts — copy and paste and it works.

If you need multiple servers, create a separate Client instance per server. Client-to-server is a 1:1 relationship.

Handling content types safely

One thing to be careful about: callTool() and readResource() return a content array where each item's structure depends on its type field.

for (const item of result.content) {
  if (item.type === "text") {
    console.log(item.text);
  } else if (item.type === "image") {
    console.log(`Image: ${item.mimeType}`);
  } else if (item.type === "resource") {
    console.log(`Resource: ${item.resource.uri}`);
  }
}

If you blindly access content[0].text without checking type, you'll get undefined when the tool returns an image or embedded resource. With third-party MCP servers, always check the type field first.

The SDK's .d.ts files define TextContent, ImageContent, and EmbeddedResource as separate types. In TypeScript, importing and using these for explicit type narrowing is cleaner than relying on runtime checks alone.

Honest limitations

A few friction points I ran into:

Thin TypeScript generics. The return type of callTool() is { content: Content[], isError?: boolean } — Content being a union type. Narrowing it to TextContent requires explicit checks. Not a blocker, but not ergonomic.

SSE/HTTP needs a different transport. For remote MCP servers (HTTP-based), you'll need StreamableHTTPClientTransport or SSEClientTransport. The setup differs slightly and isn't covered in as many examples.

Process lifecycle management. StdioClientTransport doesn't spawn a new process per call — the server process stays alive for the duration of the connection. Always call client.close() at the end of a script, or use process.on('exit', ...) to clean up, otherwise the server process lingers.

zod v4 compatibility warnings. The SDK uses zod internally, and mixing it with zod v4 in your project may produce deprecation warnings. With SDK 1.29.0 and zod 4.4.3 everything ran fine for me, but it's worth watching.

My take: the client side is underrepresented

There's a lot of content about building MCP servers. Custom client implementations — doing what Claude Desktop does programmatically — are far less documented.

The use cases are real. Developers building AI agent pipelines from scratch, teams integrating MCP into existing code, engineers debugging server behavior without a GUI. The Client class in @modelcontextprotocol/sdk is stable and the API is clean.

Seventy lines of TypeScript is enough to connect to an MCP server, list its tools, call them, read its resources, and close cleanly. I wish this had been in the official docs as a worked example from the start. Since it wasn't, here it is.

Next up: attaching this client to a real-world MCP server — probably the filesystem one or the GitHub MCP server — and building a small automation script around it.

Building AI Agents with Agno — I Actually Ran It with Gemini and Built-in Tools

Jangwook Kim — Thu, 18 Jun 2026 06:41:11 +0000

If you've ever felt like LangChain was too heavy, you're not alone. The dependency tree is enormous. Abstraction layers pile up. At some point you lose track of what's actually happening underneath. That frustration has pushed a lot of people toward lighter alternatives — frameworks that prove you can build a capable agent without a hundred transitive dependencies.

Agno is one of those alternatives. It started as Phidata and rebranded in early 2025. I spent an afternoon installing Agno v2.6.17 in a clean sandbox and running through Calculator tools, Wikipedia retrieval, Pydantic structured output, and a two-agent Team. I'll share the real execution logs and, more importantly, the traps I hit that the docs don't warn you about.

What Agno Is and Where It Came from

Phidata built a solid reputation as "the Python framework for AI assistants." When it rebranded to Agno in 2025, the design philosophy got articulated more clearly around three ideas.

Model-agnostic from day one. Over 70 LLMs — OpenAI, Anthropic, Google, Ollama, Cohere — can plug in with the same code structure. Swap the model, keep the agent logic.

Multimodal as a default. Text, image, audio, video agents all use the same API surface. You don't need a different abstraction layer for each modality.

Multi-agent orchestration as a first-class citizen. The Team class is built in. You can switch between coordinate, route, and collaborate modes with a single parameter change.

Reading that, I thought: "How is this different from LangChain?" The answer showed up when I actually wrote code. Agno favors composition over class inheritance. One agent takes about 6 lines to set up. There's far less boilerplate to wade through.

Installation: No Dependency Hell

pip install agno google-genai ddgs wikipedia

The agno package installs just the core. Tools require their own extra dependencies — wikipedia for the Wikipedia tool, google-genai for Gemini. This lazy-loading approach keeps the base install clean.

$ python3 -c "import agno; print(agno.__version__)"
2.6.17

I used the Gemini API key from my project .env. Agno auto-initializes the Gemini client from either GOOGLE_API_KEY or GEMINI_API_KEY. If both are set, it uses GOOGLE_API_KEY and prints a warning to stdout. Not a big deal, but you can't suppress it easily from code.

First Agent: Calculator Tool

from agno.agent import Agent
from agno.models.google import Gemini
from agno.tools.calculator import CalculatorTools

agent = Agent(
    model=Gemini(id="gemini-2.5-flash"),
    tools=[CalculatorTools()],
    description="A math helper agent",
)

response = agent.run("What is 2^10 + 3^5? Please use the calculator.")
print(response.content)

Output:

2^10 is 1024 and 3^5 is 243. Adding them gives 1267.
⏱ 8.98s

The math is right: 1024 + 243 = 1267. The agent invoked the Calculator tool rather than letting the LLM guess. The 9-second latency includes a Gemini API round-trip plus the tool call overhead.

Trap #1: show_tool_calls is gone. Older Agno tutorials use show_tool_calls=True. In v2.6.17, that raises TypeError: Agent.__init__() got an unexpected keyword argument 'show_tool_calls'. Use debug_mode=True instead if you want verbose output.

Trap #2: gemini-2.0-flash is deprecated. Using that model ID throws a 404:

ERROR    Error from Gemini API: 404 NOT_FOUND.
{'error': {'message': 'This model models/gemini-2.0-flash is no longer available.'}}

Use gemini-2.5-flash. Always verify the current model IDs on Google's docs before hardcoding them.

Wikipedia Agent: Automatic Search Retry

from agno.tools.wikipedia import WikipediaTools

agent = Agent(
    model=Gemini(id="gemini-2.5-flash"),
    tools=[WikipediaTools()],
)

response = agent.run(
    "What is the 'attention mechanism' in neural networks? 2 sentences only."
)

Execution log:

INFO Searching wikipedia for: attention mechanism neural networks
ERROR Error searching Wikipedia for 'attention mechanism neural networks':
      Page id "attention mechanism neural network" does not match any pages.
INFO Searching wikipedia for: attention (machine learning)
⏱ 9.98s

In machine learning, attention is a method that determines the importance
of each component in a sequence relative to the other components...

The interesting bit: the first search failed, and the agent automatically reformulated the query (attention (machine learning)) and retried. No extra code required. Agno runs a ReAct loop internally — plan, act, observe, adjust. Tool failures are handled gracefully.

Compared with the code-execution approach in Smolagents (covered in the Python AI agent library comparison post), Agno leans more toward tool composition than code generation. Neither is strictly better; it depends on what you're building.

Structured Output: Use `output_schema`, Not `output_model`

This is the most confusing naming in the API. There's a parameter called output_model. Naturally you'd think: "Put the Pydantic model here." That's wrong.

# This fails
agent = Agent(
    model=Gemini(id="gemini-2.5-flash"),
    output_model=DeveloperProfile,  # ← WRONG
)
# ValueError: Model must be a Model instance, string, or None

output_model expects an LLM model instance (or string model ID). For Pydantic structured output, use output_schema:

from pydantic import BaseModel
from typing import List

class Skill(BaseModel):
    name: str
    level: str
    year_since: int

class DeveloperProfile(BaseModel):
    name: str
    skills: List[Skill]
    summary: str

agent = Agent(
    model=Gemini(id="gemini-2.5-flash"),
    output_schema=DeveloperProfile,  # ← CORRECT
)

response = agent.run(
    "Create a developer profile for 'Kim Jangwook', a Korean developer "
    "specializing in Claude Code, MCP, Python, TypeScript."
)

print(type(response.content))   # <class '__main__.DeveloperProfile'>
print(response.content.name)    # Kim Jangwook

Actual output:

⏱ 4.00s
Type: DeveloperProfile
Name: Kim Jangwook
Skills:
  - Claude Code: Expert (since 2022)
  - MCP: Certified (since 2019)
  - Python: Senior (since 2018)
  - TypeScript: Intermediate (since 2020)

response.content returns an actual Pydantic instance. Parsing is handled internally; you get full IDE autocomplete on the result. The 4-second latency (vs 9 seconds for the Calculator agent) reflects the absence of tool call round-trips.

This is similar in spirit to PydanticAI's output_type parameter, but the naming diverges. When jumping between frameworks, you need to memorize each one's vocabulary — that's friction that accumulates.

Multi-Agent Team: `members=`, Not `agents=`

from agno.team import Team

researcher = Agent(
    name="Researcher",
    model=Gemini(id="gemini-2.5-flash"),
    tools=[WikipediaTools()],
    role="researcher",
)

calculator = Agent(
    name="Calculator",
    model=Gemini(id="gemini-2.5-flash"),
    tools=[CalculatorTools()],
    role="calculator",
)

team = Team(
    members=[researcher, calculator],  # ← NOT agents=
    model=Gemini(id="gemini-2.5-flash"),
    name="Research & Calc Team",
    mode="coordinate",
)

Trap #3: agents= doesn't exist in Team. Writing Team(agents=[...]) throws:

TypeError: Team.__init__() got an unexpected keyword argument 'agents'

The correct parameter is members=[...]. You'd only know this from reading the source — the documentation still shows agents= in some places.

Running the team:

response = team.run(
    "Find the year 'Attention is All You Need' was published on Wikipedia, "
    "then calculate how many years ago that was from 2026."
)
print(response.content)

Result:

INFO Searching wikipedia for: Attention is All You Need
⏱ 13.83s

'Attention is All You Need' was published in 2017. 
As of 2026, that is 9 years ago.

The team leader (the Gemini model passed to Team) analyzed the task, routed the Wikipedia lookup to the Researcher agent and the subtraction to the Calculator agent. Both returned correct results. The 14-second latency reflects sequential agent execution — coordinate mode doesn't parallelize in v2.6.17.

100+ Built-in Tools

One of Agno's practical advantages: the built-in tool library.

import agno.tools as t, pkgutil
tools = [name for _, name, _ in pkgutil.iter_modules(t.__path__)]
# Over 100 entries including: 'arxiv', 'bravesearch', 'calculator',
# 'docker', 'duckduckgo', 'email', 'github', 'gmail',
# 'google_bigquery', 'jira', 'mcp', 'mem0', 'notion',
# 'postgres', 'slack', 'sql', 'tavily', 'wikipedia',
# 'yfinance', 'youtube', 'zoom', ...

In practice: give Agno a BRAVE_API_KEY and you have a web search agent running in under 5 minutes without writing any API wrapper code. Same story for Slack, Notion, GitHub, and Postgres integrations.

The catch: not all tools are zero-install. Each tool module has its own dependency. agno.tools.duckduckgo needs ddgs, agno.tools.wikipedia needs wikipedia, and so on. If you import before installing, you get ImportError at the import line for some tools and at first use for others. Inconsistent behavior across modules.

What I Think Are the Actual Limitations

The latency is real. 9 seconds for a single Calculator call, 14 for a two-agent team — this is Gemini API round-trip cost compounded by tool calls, not an Agno inefficiency per se. But it matters for production APIs where users expect sub-second responses.

Debugging is manual. debug_mode=True spits out unstructured logs. I haven't found an official integration guide for LangSmith or LangFuse. If observability matters, you'll need to wire it up yourself.

The docs lag behind the API. show_tool_calls, output_model, agents= — these are examples of parameters whose behavior in the docs doesn't match the current codebase. Always check the GitHub examples/ directory for the latest version, not the tutorial blog posts.

The Team's coordinate mode is sequential. If you need parallel agent execution or complex conditional branching across many agents, Google ADK or LangGraph are better fits.

When Agno Makes Sense

Three scenarios where I'd reach for Agno:

Rapid prototyping. An API key plus 10 lines of Python and you have a working agent. Great for PoCs, internal tools, solo projects where speed to first working version matters more than architectural elegance.

Multi-tool agents. When you need an agent that touches Slack, reads from Postgres, sends an email, and searches the web — Agno's tool library means you spend time on the agent logic, not on writing API wrappers.

Small agent teams (2-4 agents). Agno's Team class handles small coordination problems cleanly. Once you get into dozens of agents with complex dependency graphs, a state-graph framework gives you more explicit control.

Where I wouldn't use Agno: real-time streaming UIs, production workflows requiring precise error handling and retry guarantees, or systems where you need to audit exactly what each agent decided and why at every step.

What's Next to Explore

Two things I didn't test today:

Agent memory. Agno has enable_agentic_memory=True with SQLite-backed storage. Cross-session memory persistence is the piece that would make agents feel genuinely stateful rather than starting fresh each time.

MCP tool integration. agno.tools.mcp exists. If Agno agents can connect to MCP servers as tool sources, that means reusing existing MCP server infrastructure without rewriting anything. Worth testing.

Summary

Agno v2.6.17 installs cleanly via pip install agno; Calculator, Wikipedia, Team all ran successfully
Use gemini-2.5-flash as the model ID — gemini-2.0-flash is deprecated and returns 404
Structured output uses output_schema=YourPydanticModel, not output_model
Team takes members=[...], not agents=[...]
100+ built-in tools, each requiring its own dependency install
Strong for prototyping and multi-tool agents; reach for LangGraph for complex state machines

If you've been frustrated by LangChain's weight and want a Python agent framework that gets out of your way, Agno is worth an afternoon of experimentation.

Ollama Structured Outputs in Practice — Getting Type-Safe JSON from Local LLMs with Pydantic

Jangwook Kim — Wed, 17 Jun 2026 06:38:48 +0000

json.loads(response) fails at a certain point. You told the model "return JSON only," but it added a

```json markdown code fence around everything. A quick regex strips it — until that regex hits an edge case, and that edge case blows up in production.

Since Ollama 0.3.0, passing a JSON schema to the format parameter eliminates this problem at the root. The model's inference itself is constrained by the schema, so no code fences, no explanatory text, no mid-thought artifacts. Just parseable JSON.

I ran these tests locally with Gemma4 and Ollama 0.30.7 to see how well it holds up in practice.

Why LLM Response Parsing Is Tricky

The most common problem when running Ollama locally — without a cloud LLM API — is JSON parsing. Two reasons.

First, text generation models are trained toward "natural text." Even if you ask for JSON only, they'll often wrap it in json ... blocks or prepend "Of course! Here is the JSON you requested:" style text. Here's what I reproduced directly:


json
Input: 'Give me 3 Python tips as JSON with keys: tips (array), difficulty (1-5)'
Model output (no format parameter):


```json
{
  "tips": [
    "Master the fundamentals first...",
    ...
  ]
}

JSON parse: FAILED


Python's `json.loads()` can't handle the markdown wrapper. The "JSON only" instruction is unreliable in production.

Second, speed. I measured the same query both ways: 32 seconds without structured output, 5 seconds with it. More on why below.

## How the Ollama format Parameter Works

Ollama's `/api/generate` endpoint has a `format` field. Pass a JSON schema object and Ollama applies **constrained decoding** during inference.

python
import json
import urllib.request

def ollama_structured(prompt, schema, model="gemma4:e4b"):
payload = {
"model": model,
"prompt": prompt,
"format": schema, # ← pass JSON schema object directly
"stream": False,
"options": {"temperature": 0}
}
data = json.dumps(payload).encode()
req = urllib.request.Request(
"http://localhost:11434/api/generate",
data=data,
headers={"Content-Type": "application/json"}
)
with urllib.request.urlopen(req, timeout=60) as resp:
result = json.loads(resp.read())
return result["response"]


Constrained decoding sets the probability of any token that would violate the schema to zero at each generation step. So even if the model "wants" to generate a markdown fence, the schema makes it physically impossible. That's also where the speed gain comes from — the model doesn't waste tokens on formatting decisions.

Here are the measured numbers:

bash

Direct measurement (Ollama 0.30.7, Gemma4:e4b, MacBook)

Same prompt, with and without format

Without structured output:
Raw (first 200 chars):

```json\n{\n "tips": ["Master the fundamentals first...
Time: 31.84s
JSON parse: FAILED (markdown wrapper)

With structured output:
Structured: {"tips": ["Understand the concept of indentation...", ...], "difficulty": 2}
Time: 4.99s
JSON parse: SUCCESS




6.4x difference. Local LLMs are already slow, and adding unreliable parsing on top makes the whole pipeline feel worse.

## Wiring Pydantic Models

Writing JSON schema objects by hand is tedious. With Pydantic models, `model_json_schema()` generates the schema automatically.



```python
from pydantic import BaseModel
from typing import List, Dict, Any, Literal

class CodeReview(BaseModel):
    severity: str  # "critical", "warning", "info"
    file: str
    line: int
    message: str
    suggestion: str

class ReviewResult(BaseModel):
    total_issues: int
    critical_count: int
    reviews: List[CodeReview]

# Pydantic → JSON schema, automatically
schema = ReviewResult.model_json_schema()

raw = ollama_structured(prompt, schema)

# Parses and validates in one step
result = ReviewResult.model_validate_json(raw)

model_validate_json parses the JSON string and runs Pydantic validation simultaneously. If severity gets an integer or line gets a string, it throws ValidationError. Catching that and retrying with a modified prompt is the common pattern in real agents.

Actual output from the code review test:

=== Code Review Output ===
Total issues: 3
Critical: 2
  [CRITICAL] process_user_data:2 - SQL Injection Vulnerability (Direct String Formatting)
  [CRITICAL] process_user_data:3 - Storing Passwords in Plain Text (Data Leakage)
  [HIGH] process_user_data:4 - Potential Unused/Incomplete Database Interaction

total_issues: 3 and critical_count: 2 come in as integers. if result.critical_count > 0 branches safely.

Practical Pattern: Agent Tool Dispatch

The strongest use case for structured output is an agent deciding which tool to call next. You pass the tool list and current situation, and get back a type-safe tool call selection.

from typing import Literal, Dict, Any

class ToolCall(BaseModel):
    tool_name: Literal["web_search", "read_file", "write_file", "execute_code"]
    parameters: Dict[str, Any]
    reasoning: str

schema = ToolCall.model_json_schema()

user_task = "Find the current Bitcoin price and save it to btc_price.txt"
prompt = f"""You are an AI agent. Decide which tool to call next.
Available tools: web_search, read_file, write_file, execute_code
Task: {user_task}
Choose ONE tool call."""

raw = ollama_structured(prompt, schema)
tool_call = ToolCall.model_validate_json(raw)

print(f"Tool: {tool_call.tool_name}")
print(f"Params: {tool_call.parameters}")

=== Agent tool dispatch ===
Tool: web_search
Params: {'query': 'current Bitcoin price'}
Reasoning: The task requires finding real-time information...

Dispatch: OK (type-safe)

Because tool_name is typed as Literal["web_search", "read_file", ...], tool_call.tool_name is always one of those four values. If the model invents a nonexistent tool name, Pydantic throws ValidationError. The if tool_call.tool_name == "web_search" branch is safe to write.

This is architecturally the same as function calling in cloud APIs. Comparing it with Claude Agent SDK's Tool Use patterns shows an interesting design difference: cloud LLMs handle tool selection natively at the model level, while local Ollama needs an explicit JSON schema + Pydantic validation layer.

Gemma4 and Schema Complexity: Limitations I Found

Honestly, it doesn't work perfectly in every case. Testing with Gemma4:e4b (4-bit quantized, 4B parameters), I found a few real constraints.

Deeply nested schemas. JSON schemas nested 3+ levels deep (List[Dict[str, List[BaseModel]]]) sometimes return empty arrays at intermediate levels. The 12B model (gemma4:12b-it-qat) reduces this, but doesn't eliminate it. This is a fundamental limitation of the model's context handling.

Optional field handling. Fields declared as Optional[str] sometimes get filled with empty string "" instead of null. Pydantic validation passes, but semantics differ. You need @validator post-processing.

Schema size. A large Pydantic model's JSON schema can reach hundreds of tokens. That occupies context window space, reducing the room available for the actual prompt. Complex schemas need stronger models.

Once you've deployed Ollama as an API server (covered in the Ollama FastAPI production guide), switching models at runtime based on schema complexity becomes a viable optimization.

Pattern Reference: When to Use What

Situation	Approach	Why
Simple data extraction (1-2 levels)	`format` + `json.loads()`	Fast, no overhead
Type validation needed	`format` + Pydantic	ValidationError catches issues early
Agent tool selection	`format` + Pydantic `Literal`	Blocks invalid tool names
Complex nested schema	Consider larger model	Small local model limitations
Simple text response	No `format`	Avoid unnecessary constrained decoding overhead

I think of this as a switch that moves JSON parse reliability from "unreliable" to "near 100%." There was a time I was appending "JSON only please" to every prompt and hoping for the best. Measuring the actual difference made clear how fragile that approach was.

Copy-Paste Starter Code

import json
import urllib.request
from typing import List, Optional, Dict, Any
from pydantic import BaseModel

def ollama_structured(prompt: str, model_cls: type[BaseModel], 
                      model: str = "gemma4:e4b") -> BaseModel:
    """
    Helper that combines Ollama structured output + Pydantic validation.
    """
    schema = model_cls.model_json_schema()
    payload = {
        "model": model,
        "prompt": prompt,
        "format": schema,
        "stream": False,
        "options": {"temperature": 0}
    }
    data = json.dumps(payload).encode()
    req = urllib.request.Request(
        "http://localhost:11434/api/generate",
        data=data,
        headers={"Content-Type": "application/json"}
    )
    with urllib.request.urlopen(req, timeout=120) as resp:
        result = json.loads(resp.read())
    return model_cls.model_validate_json(result["response"])


# Usage example
class SentimentAnalysis(BaseModel):
    sentiment: str       # "positive", "negative", "neutral"
    confidence: float    # 0.0 ~ 1.0
    key_phrases: List[str]

result = ollama_structured(
    "Analyze sentiment: 'This new MacBook is amazing but too expensive'",
    SentimentAnalysis
)
print(f"Sentiment: {result.sentiment} ({result.confidence:.0%})")
print(f"Key phrases: {result.key_phrases}")

What to Try Next

This only covers the simplest cases. A real agent needs a bit more.

Retry logic. When Pydantic ValidationError fires, retry with a slightly modified prompt — ideally including the error message. Models often self-correct when they can see why they were wrong.

Streaming. With stream: true, you can receive the JSON incrementally as it generates. Pair with a streaming JSON parser like ijson for memory-efficient handling of large responses.

Model switching. Route simple extractions to gemma4:e4b (fast) and complex nested schemas to gemma4:12b-it-qat (accurate) at runtime. Structuring an entire agent with Pydantic AI shows how to abstract this decision to the framework level.

If you're already running a Gemma4-based agent locally, adding the format parameter today is a one-line change with a measurable reliability improvement. Especially anywhere in the agent loop where an invalid response immediately causes a downstream error.

Testing RAG Embeddings Hands-On with sentence-transformers — Why Korean Queries Drop Accuracy by 67%

Jangwook Kim — Tue, 16 Jun 2026 06:41:12 +0000

When I first learned about RAG, embeddings were an abstraction I accepted without questioning. "Sentences get converted to vectors," "similar meanings end up close together in vector space" — all true, but none of it clicked until I actually measured the numbers. So I installed sentence-transformers locally, measured cosine similarities, ran a mini retrieval simulation, and checked what happens with Korean queries specifically.

The short answer: building a Korean RAG system with an English-optimized embedding model dropped accuracy by 67% in my tests. Two out of three queries returned the wrong document at rank 1. This post documents that experiment and how switching to a multilingual model fixed it.

What it actually means to run an embedding model locally with pip

I didn't believe it was possible without a cloud API at first. OpenAI and Gemini embeddings require API keys and round-trips to external servers. But sentence-transformers just downloads model weights locally.

pip install sentence-transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("all-MiniLM-L6-v2")
embedding = model.encode("building an AI agent")
print(embedding.shape)  # (384,)

all-MiniLM-L6-v2 downloads from Hugging Face Hub on first run. Load time on my machine: 9.52 seconds. After that, it's cached and loads in under 1 second. Model size is around 22MB — genuinely lightweight.

The vector internals are worth inspecting:

Type: numpy.ndarray
Shape: (384,)
dtype: float32
L2 norm: 1.000000
Min: -0.147123
Max: 0.183166
First 5 dims: [-0.0216, 0.0593, -0.0049, -0.0172, 0.0079]

The L2 norm being exactly 1.0 is interesting. sentence-transformers normalizes embeddings by default, which means cosine similarity and dot product return identical results. This is a nice property — you don't need to remember to normalize before computing similarity.

384 dimensions is a deliberate tradeoff. Modern large embedding models use 1536〜3072 dimensions, which is more expressive but costs more to store and search. For most semantic search use cases, 384 is plenty.

Cosine similarity: what the numbers actually look like

from sentence_transformers.util import cos_sim

pairs = [
    ("AI agent uses tools to complete tasks",
     "An autonomous agent invokes functions to achieve goals"),
    ("AI agent uses tools to complete tasks",
     "The cat sat on the warm mat by the window"),
    ("How do I install Python packages?",
     "What is the command to add a Python library?"),
    ("How do I install Python packages?",
     "The stock market closed higher today"),
]

Measured similarity scores:

0.6489 ████████████  AI agent vs autonomous agent (same meaning, different words)
-0.0112            AI agent vs cat on mat (unrelated)
0.6248 ████████████  install packages vs add library (same meaning)
-0.0163            install packages vs stock market (unrelated)

Semantically similar pairs land at 0.62〜0.65. Unrelated pairs are near zero or slightly negative. The negative values surprised me at first — cosine similarity ranges from -1 to 1, so truly unrelated sentences naturally cluster around 0, sometimes dipping slightly negative.

For context on what "high" looks like in practice: the vector DB benchmark post notes that RAG systems typically use 0.3〜0.5 as a retrieval threshold. So 0.65 is a strong semantic match.

Mini RAG simulation — two out of three queries failed

I ran a retrieval simulation with 10 Korean blog post titles as the knowledge base and 3 Korean queries. Model: English-optimized all-MiniLM-L6-v2.

Knowledge base included titles like:

"Reduce LLM costs 70% with Claude API Prompt Caching"
"MCP vs A2A vs Open Responses — agent protocol comparison"
"Use Node.js built-in SQLite without external packages"

Queries:

"How do I reduce LLM API costs?"
"Compare agent-to-agent communication protocols"
"Lightweight database without Python"

Results:

Query: "How do I reduce LLM API costs?"
  #1 [0.5649] Anthropic Message Batches API for bulk processing  ← expected: Prompt Caching
  #2 [0.4534] Claude API Prompt Caching — 70% cost reduction

Query: "Compare agent-to-agent communication protocols"
  #1 [0.5605] AI Agent Observability Guide  ← completely wrong topic
  (MCP vs A2A vs Open Responses ranked outside top 3)

Query: "Lightweight database without Python"
  #1 [0.5172] LangGraph multi-agent state management  ← irrelevant
  (Node.js SQLite not in top 3)

One query found the right answer at rank 2. The other two were completely off. The "agent protocol comparison" failure is particularly bad — there's a document with "protocol comparison" literally in its title, and the model ranked an observability guide above it. This is what an English model mishandling Korean semantics looks like.

Why English models fail on Korean and how multilingual models fix it

all-MiniLM-L6-v2 comes from the SBERT paper and was trained primarily on English sentence pairs. It can process Korean text but doesn't represent Korean semantics with the same fidelity as English.

paraphrase-multilingual-MiniLM-L12-v2 was trained on parallel corpora across 50+ languages. It's explicitly designed to place the same meaning in different languages close together in vector space.

Same 3 queries, both models:

Query	English model #1 (correct?)	Multilingual #1 (correct?)
Reduce LLM API costs	Message Batches [0.453] ✗	Prompt Caching [0.720] ✓
Agent protocol comparison	Observability [0.561] ✗	MCP vs A2A [0.647] ✓
Lightweight DB without Python	LangGraph [0.461] ✗	Node.js SQLite [0.439] ✓

English model: 1/3 (33%). Multilingual: 3/3 (100%).

The similarity scores are also telling. The multilingual model assigned 0.720 to the correct cost-reduction document; the English model only managed 0.453 for the same pair. Stronger semantic connection, better ranked.

The RAG architecture post argues that retrieval quality determines generation quality. This experiment shows that principle applies at the embedding model selection step — before you've written a single line of retrieval code.

My conclusion: for Korean or multilingual RAG pipelines, start with a multilingual model. Switching later means re-embedding your entire document collection, which is a meaningful operational cost for large corpora.

Batch encoding: the 2.4x throughput gap

Sequential encoding is the obvious first implementation. Batch encoding is faster for reasons that have to do with CPU/GPU parallelism and memory bandwidth — but how much faster exactly?

sentences = [f"This is test sentence number {i}" for i in range(100)]

# Sequential: 1.075 seconds
for s in sentences:
    model.encode(s)

# Batch: 0.455 seconds
model.encode(sentences, batch_size=32, show_progress_bar=False)

Sequential (100 sentences): 1.075s
Batch (100 sentences):      0.455s → 2.4x faster
Throughput (batch):         220 sentences/sec

220 sentences/second on CPU. For 10,000 documents at 5 sentences each, that's 50,000 sentences — about 227 seconds, under 4 minutes. With GPU (CUDA or Apple Silicon MPS), throughput reportedly scales 10〜50x higher, though I didn't test that directly.

The practical implication: use model.encode(batch, batch_size=N) instead of calling encode in a loop. This is especially true for initial indexing jobs.

Putting this into an actual RAG pipeline

Today's experiment covered the R in RAG. A complete pipeline looks like:

# Step 1: Index documents (offline, run once)
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")
documents = load_your_documents()
doc_embeddings = model.encode(documents, batch_size=32)
# Store in a vector DB (Chroma, Qdrant, pgvector)

# Step 2: Retrieve at query time (online)
query_embedding = model.encode(user_query)
# Fetch top-k by cosine similarity from vector DB

# Step 3: Generate with context (online)
context = "\n".join(retrieved_docs)
# Pass context to Claude or another LLM

One practical detail: store the model name as metadata alongside your embeddings. If you ever change models, you need to re-embed everything. Without metadata tracking, you risk mixing vectors from different embedding spaces in the same database — a subtle bug that produces silently degraded retrieval.

What I still don't know and what's next

Limitation 1: My knowledge base had only 10 documents. Real retrieval performance over thousands of documents could look quite different, and I don't have that data yet.

Limitation 2: 3/3 accuracy for the multilingual model is based on 3 test cases. Not statistically meaningful. I'd need hundreds of test pairs to make a confident claim.

Limitation 3: Embeddings have a known weakness: they struggle when specific keywords matter more than semantic similarity. "Without Python" as a constraint is the kind of thing BM25 handles better than dense retrieval. Hybrid search (BM25 + embeddings via reciprocal rank fusion) is the standard solution here, but I haven't implemented it yet.

What's next: testing multilingual-e5-large on the same benchmark, wiring up a persistent ChromaDB store, and measuring hybrid search accuracy against pure vector retrieval.

I went into this experiment thinking embedding models were interchangeable plug-ins. The 67% accuracy gap changed that view. For non-English RAG, model selection is a first-order decision that affects the entire pipeline downstream.

Mastra.ai Practical Guide — Running a TypeScript AI Agent in 5 Minutes

Jangwook Kim — Sun, 14 Jun 2026 06:42:49 +0000

"If you're a TypeScript developer building AI agents, you're stuck with LangChain.js or Vercel AI SDK." I've heard that line a lot. It felt accurate enough that I never seriously questioned it — until Mastra.ai popped up on my radar.

Mastra hit v1.0 in January 2026 after graduating from YC W25 with $13M in funding. By then it had already passed 22,000 GitHub stars and 300,000 weekly npm downloads. The Gatsby.js team built it, which means it comes from people who understand what "framework" should actually mean for JavaScript developers.

I had heard the name. Today I finally installed it, wired it to Google Gemini, and ran an actual agent with tool calls. This is the record of what worked, what didn't, and whether the "5-minute agent" claim holds up.

What Mastra.ai Actually Is

Mastra is a TypeScript-first framework that bundles agents, workflows, memory, and observability into a single SDK. The pitch is simple: stop gluing packages together. Define your agent, its tools, and its memory in one place, and let the framework handle the plumbing.

It supports any LLM provider that Vercel AI SDK covers — OpenAI, Anthropic, Google Gemini, Meta Llama, and more. Mastra uses Vercel AI SDK as its underlying layer, so if you've built a streaming Claude agent with Vercel AI SDK before, you're already familiar with the foundation Mastra sits on.

Why Now?

The Python agent ecosystem got mature fast. LangGraph, CrewAI, PydanticAI — these tools accumulated years of production use, community plugins, and battle-tested patterns. The TypeScript side lagged. Mastra is an attempt to close that gap, and v1.42 (the version I installed today) suggests it's serious about catching up.

I expected to be disappointed. I wasn't.

Installation: One Command

I followed the official quickstart. Node.js v22.22.0 on my machine.

npm create mastra@latest mastra-lab -- --components agents,tools --llm google --example

The flags: --components agents,tools pulls in the agent and tool scaffolding, --llm google sets Google Gemini as the provider, --example generates a working weather agent template.

Setup takes 2〜3 minutes. The CLI walks through each step:

◇  Project structure created
◇  npm dependencies installed
◇  Mastra CLI installed
◇  Mastra dependencies installed
◇  .gitignore added
└  Project created successfully

◇  Mastra initialized successfully!

   Rename .env.example to .env
   and add your GOOGLE_API_KEY

Core dependencies in the generated package.json:

{
  "dependencies": {
    "@mastra/core": "^1.42.0",
    "@mastra/memory": "^1.20.3",
    "@mastra/libsql": "^1.13.0",
    "@mastra/observability": "^1.14.1",
    "zod": "^4.4.3"
  },
  "devDependencies": {
    "typescript": "^6.0.3"
  }
}

TypeScript 6.0.3 and Zod v4. Both bumped major versions in early 2026. The fact that Mastra ships with current versions of both is a good signal.

Project Structure

mastra-lab/
├── src/
│   └── mastra/
│       ├── index.ts          # Mastra instance setup
│       ├── agents/
│       │   └── weather-agent.ts  # Agent definition
│       └── tools/
│           └── weather-tool.ts   # Tool definition
├── .env.example
├── package.json
└── tsconfig.json

The separation is clean. agents/ holds agent definitions. tools/ handles external API interfaces. index.ts wires everything together into a Mastra instance.

The Code: Tools and Agents

Looking at the generated code reveals Mastra's design priorities.

Tool Definition

// src/mastra/tools/weather-tool.ts
import { createTool } from '@mastra/core/tools';
import { z } from 'zod';

export const weatherTool = createTool({
  id: 'get-weather',
  description: 'Get current weather for a location',
  inputSchema: z.object({
    location: z.string().describe('City name'),
  }),
  outputSchema: z.object({
    temperature: z.number(),
    feelsLike: z.number(),
    humidity: z.number(),
    windSpeed: z.number(),
    conditions: z.string(),
    location: z.string(),
  }),
  execute: async (inputData) => {
    return await getWeather(inputData.location);
  },
});

Zod schemas for I/O typing. If you've seen PydanticAI's type-safe agent approach in Python, this is structurally similar — "type definition as documentation and validation logic" across different languages.

The weather tool uses Open-Meteo, which is free and needs no API key. It geocodes city names to coordinates, then fetches current weather data. Clean separation of concerns.

Agent Definition

// src/mastra/agents/weather-agent.ts
import { Agent } from '@mastra/core/agent';
import { Memory } from '@mastra/memory';
import { weatherTool } from '../tools/weather-tool';

export const weatherAgent = new Agent({
  id: 'weather-agent',
  name: 'Weather Agent',
  instructions: `You are a helpful weather assistant...`,
  model: 'google/gemini-2.5-pro',
  tools: { weatherTool },
  memory: new Memory(),
});

The model string 'google/gemini-2.5-pro' is all Mastra needs to wire up the right provider. @ai-sdk/google handles the actual API calls underneath.

Running It: Seoul vs Tokyo Weather

I ran the agent directly without going through the full Mastra server setup. Memory was excluded initially — more on why below.

import { Agent } from '@mastra/core/agent';

const agent = new Agent({
  id: 'weather-agent',
  name: 'Weather Agent',
  instructions: 'Provide concise weather information.',
  model: 'google/gemini-2.5-flash',
  tools: { weatherTool },
});

const result = await agent.generate(
  'Compare the current weather in Seoul and Tokyo. Which city is hotter right now?'
);
console.log(result.text);

Output (2026-06-14, response time: 5,866ms):

It is 27.3°C and feels like 30.1°C in Seoul with mainly clear conditions.
In Tokyo, it is 25.6°C and feels like 27.1°C with overcast conditions.
Seoul is hotter right now.

The agent called get-weather twice — once for Seoul, once for Tokyo — synthesized the results, and answered the comparison question. Two external API calls plus LLM reasoning in under six seconds. Seoul was indeed hotter.

Location resolution worked correctly too. "Seoul" mapped to coordinates 37.566, 126.978 without any prompting.

The Error I Hit: Memory Needs Storage

My first attempt included memory: new Memory() in the agent definition. This errored immediately:

MastraError: Memory requires a storage provider to function.
Add a storage configuration to Memory or to your Mastra instance.
https://mastra.ai/en/docs/memory/overview

The official example includes Memory, but Memory needs a storage backend — LibSQL or DuckDB. The generated index.ts has this configured, but when running an agent standalone, that configuration is missing.

This is a real friction point for new users. The error message points to docs, but someone just trying to run the quickstart example will hit this and wonder what went wrong. Better scaffolding here would help.

Architecture: Four Layers

Mastra organizes around four layers:

1. Agent Layer

Calls the LLM and decides whether to invoke tools. A single generate() call can involve multiple internal LLM↔tool round trips.

2. Tools/Integrations

Zod-typed interfaces to external APIs, databases, or any other service. The LLM fills in arguments according to the schema.

3. Memory

Conversation history, semantic search, and working memory. Requires LibSQL or PostgreSQL as backing store.

4. Observability

OpenTelemetry-based tracing, spans, and logging for every agent execution. MastraStorageExporter persists locally; MastraPlatformExporter sends to Mastra's cloud platform.

Mastra Studio

Running npm run dev opens Mastra Studio at http://localhost:4111. It's a web UI for chatting with agents, inspecting tool call traces, and testing workflows. During development, it's genuinely useful — you can see exactly which tools fired and what they returned, without digging through logs.

How It Compares

The closest comparison in TypeScript is Vercel AI SDK. Vercel AI SDK handles LLM calls and streaming well; Mastra adds agent lifecycle management, memory, and observability on top. It's not a replacement — it's a higher-level abstraction that uses AI SDK underneath.

Against the Google ADK vs LangGraph comparison I did earlier, both were Python-only. Mastra is occupying that same space in TypeScript.

	Mastra	Vercel AI SDK	LangGraph.js
Language	TypeScript	TypeScript	TypeScript
Agent loop	✅ Built-in	⚠️ Manual	✅ Built-in
Memory	✅ Built-in (storage req'd)	❌	⚠️ Manual
Workflows	✅ Graph-based	❌	✅ Graph-based
Observability	✅ OpenTelemetry	❌	⚠️ External tools
Learning curve	Medium	Low	High

What I'd Change

Two things stood out as friction:

The Memory setup barrier. New users following the official example will hit an error the first time they try to use Memory standalone. The error is informative but the setup path isn't obvious. A cleaner getting-started experience here would make a real difference.

Production deployment documentation is thin. Mastra Studio is a development tool, but the path from "working locally" to "deployed on my server" isn't well documented outside of the Vercel deployment guide. Docker and self-hosted setups require figuring things out yourself.

Is It Worth Trying Right Now?

Yes, with caveats.

If you're starting a new TypeScript agent project, Mastra is worth a serious look. The setup is fast, the structure is sensible, and the core abstractions hold up. It took me about ten minutes from nothing to a working agent with real tool calls — the "5-minute" claim is close to accurate.

For production: I'd wait. Mastra v1.0 shipped in January 2026. Six months of production exposure is not much. The ecosystem is still thin compared to LangChain. API stability is not yet something I'd bet a production service on.

For a side project or internal tool: use it now. For a production service: revisit around v1.5.

# Get started
npm create mastra@latest my-agent-app -- --components agents,tools --llm google --example
cd my-agent-app
# Add GOOGLE_API_KEY to .env
npm run dev
# → Open http://localhost:4111 to chat with your first agent

The TypeScript AI agent ecosystem finally has a real contender. It's not fully there yet, but it's the most promising thing I've seen on the TypeScript side in a while.

Anthropic and OpenAI Filed for IPO in the Same Month — What the Token Price War Means for Developers

Jangwook Kim — Sat, 13 Jun 2026 06:41:31 +0000

On June 1st, news broke that Anthropic had quietly filed a confidential S-1 registration statement with the SEC. One week later, on June 8th, OpenAI did the same thing. Two of the most important companies in AI filing for IPO in the same month — that's never happened before in this space, and it's not just a capital markets story. It has real implications for how much developers pay to build with these models.

The short version: this is good for you right now

Let me be direct. In the short term, this competitive dynamic works in developers' favor. Both companies need to demonstrate strong growth metrics before they go public, and one lever they can pull is lowering prices to attract more API usage. OpenAI is reportedly considering "drastic" cuts to token prices. Anthropic restructured toward consumption-based revenue earlier this year, meaning their ARR grows the more developers use the API.

The concern I have is about what happens after the IPO. Once there are public shareholders to answer to, the pressure on margins increases. And the competitive threat from Chinese open-source models, which is driving much of this pricing discussion, doesn't disappear once either company is listed.

What actually happened in the first week of June

Anthropic filed a confidential draft S-1 with the SEC on June 1, 2026. The confidential process lets the SEC review before the full prospectus goes public. Here's what we already know:

Anthropic's most recent Series H: $65 billion raised
Post-money valuation from that round: approximately $965 billion
Annualized revenue run rate (ARR): around $47 billion per reports
Likely IPO timeline: as early as October 2026 on Nasdaq or NYSE

OpenAI followed on June 8, with a valuation around $852 billion based on its March 2026 raise — at that point, lower than Anthropic for the first time.

The timing isn't coincidental. Both companies are chasing the same pool of institutional investors while AI enthusiasm is still high. If one company lists first and locks up investor attention, the other's roadshow gets harder. The race to file is, in part, a race to get on the public markets before the window closes.

Why the IPO race is creating pricing pressure right now

To build a compelling IPO story, you need revenue, growth rate, and a credible market position narrative. Right now, both companies are being squeezed on that third point — not by each other, but by Chinese open-source models.

DeepSeek V3.2 delivers GPT-5-level coding performance at $0.28 input / $0.42 output per million tokens. That's roughly 10x cheaper than Claude Sonnet 4.6 ($3.00/$15.00) on the input side, and 18x cheaper than Opus 4.8 ($5.00/$25.00). Qwen3-Max sits in a similar price range. If enterprise customers can get comparable quality at that price, there's a real pull to route non-sensitive workloads there.

I covered Anthropic's earlier pricing shift away from third-party subscriptions in April. That move was partly about converting OpenClaw and similar tools from cheap subscription access to consumption billing. The logic is the same now: the more API tokens developers burn, the better the ARR story looks heading into an IPO roadshow.

What Anthropic's IPO financials actually reveal

The ~$47B ARR figure that's been reported isn't just a big number — it tells you something about how Anthropic has been building its revenue structure heading into a public offering.

Anthropic has three meaningful revenue streams now: API consumption, enterprise contracts, and Claude.ai subscriptions. The fastest-growing channel is API consumption. This is directly connected to the strategic moves we saw earlier in the year, including blocking third-party subscription access through products like OpenClaw. When you cut off cheap subscription API routing and force developers onto direct API billing, every token burned shows up in the ARR.

For IPO purposes, what investors care about isn't just current revenue — it's revenue growth rate and its predictability. If Anthropic's ARR was $20B six months ago and is now $47B, that's a 2.35x multiplier over six months. Maintaining that trajectory into the IPO roadshow is worth a lot. And one way to sustain it is to lower prices just enough to increase volume without tanking margins.

This means the price discounts developers are seeing right now are a feature, not a bug, of Anthropic's IPO strategy. I'm not saying that to be cynical — it's genuinely useful to understand the incentive structure. Anthropic wants high API usage because it shows the market that Claude is winning.

What the current API costs actually look like

I installed @anthropic-ai/sdk 0.104.1 in a sandbox and worked through some token cost scenarios. The numbers below are based on official published pricing as of June 13, 2026. OpenAI's is pre-any announced cut.

Caching changes things meaningfully. When prompt cache hits, input token cost drops to 10% of the standard rate (a 90% discount). For a 50K input + 2K output scenario:

Without caching: Claude Sonnet 4.6 → $0.18 / Claude Haiku 4.5 → $0.06
With 80% cache hit rate: Claude Sonnet 4.6 → ~$0.072 / Claude Haiku 4.5 → ~$0.024

If you're repeatedly sending large codebases or long system prompts, effective costs can come down to 30-40% of sticker price. The Claude prompt caching optimization guide has examples of 70% real-world reductions. That's meaningful — though it still doesn't close the full gap with DeepSeek at scale.

Looking at Claude Fable 5's $10/$50 pricing next to these other tiers, you can see Anthropic is running a dual strategy: premium pricing for frontier capability (Fable 5, Opus 4.8) while keeping Haiku competitive on the low end. The competitive pricing pressure probably won't touch Fable 5 much — workloads that need that level of performance tend to be price-inelastic. It's Sonnet and Haiku where the real price competition happens.

If OpenAI cuts GPT-5.4 from $2.50 to $1.50, that puts Sonnet 4.6 at $3.00 in an awkward position. Anthropic would likely respond. That's the scenario that would actually benefit developers most, and it becomes more likely the closer both companies get to their respective IPOs.

The risks developers should actually worry about post-IPO

I'll be candid here: I think the current pricing pressure is in large part a pre-IPO phenomenon. There are structural reasons to expect it looks different afterward.

Shareholder pressure changes the calculus. Private companies can sustain losses to build market share. Public companies face quarterly margin scrutiny. AWS, Azure, and GCP all went through a period of aggressive pricing before consolidating around higher margins once they dominated their markets. There's no reason to assume Anthropic or OpenAI won't follow a similar arc.

Lock-in is accumulating. Claude Code, Cursor, Windsurf, and other AI coding tools are increasingly built around Claude. The more your workflows are optimized for a specific model's behavior, the higher the cost of switching when pricing changes. I've been watching this — the June 2026 Claude Code update added features that deepen that integration further.

The Chinese model data problem isn't going away. DeepSeek might be 10-30x cheaper, but routing your proprietary codebase or customer data through Chinese infrastructure is a conversation most enterprise legal and security teams won't approve. EU GDPR, US defense sector regulations, healthcare — all of these narrow the viable options. Anthropic and OpenAI carry enterprise trust that open-source models can't easily replicate.

What I'm doing about it

Three things I've actually changed in how I build:

First, I'm keeping model selection in config, not code. Every place where I used to write claude-sonnet-4-6 directly, I've moved to environment variables or config files. When pricing changes or a better option appears, switching costs drop significantly.

// Before: model name baked in
const msg = await client.messages.create({
  model: "claude-sonnet-4-6",
  // ...
});

// After: externalized, switchable without code changes
const MODEL = process.env.CLAUDE_MODEL ?? "claude-sonnet-4-6";
const msg = await client.messages.create({
  model: MODEL,
  // ...
});

This looks trivial but matters when you're running workflows across multiple services. Changing one environment variable rather than searching a codebase is the difference between a five-minute update and a half-day refactor.

Second, I've prioritized prompt caching implementation. For anything involving repeated large context — codebase analysis, document review, multi-turn agents — the 90% cache discount is the fastest win available. Waiting for prices to drop while ignoring this seems backward.

const msg = await client.messages.create({
  model: MODEL,
  system: [
    {
      type: "text",
      text: largeSystemPrompt,  // Your long context goes here
      cache_control: { type: "ephemeral" }  // One line to activate caching
    }
  ],
  messages: [{ role: "user", content: userInput }],
  max_tokens: 2048,
});

The cache_control field is all it takes. If your system prompt is 50K tokens and you're making 100 requests per hour, you've just saved 90% on 50K × 100 = 5M tokens worth of input cost per hour. Do the math for your actual usage — for many codebases-as-context use cases, caching alone makes Anthropic competitive with cheaper alternatives.

Third, I'm routing workloads by risk profile. Tasks that don't touch internal code or customer data are candidates for DeepSeek or Qwen testing. Tasks that do, stay on Anthropic or OpenAI. It's not a value judgment on the models — it's practical risk management.

My current decision rule is simple: does this request contain customer names, internal source code, or business strategy? If yes, Anthropic or OpenAI. If it's processing public technical documentation, general text summarization, or analysis of public data, DeepSeek goes on the testing list. Following this rule alone can reduce monthly costs 20-30% for many teams without creating meaningful compliance exposure.

My take: you're a price war beneficiary, but stay clear-eyed about the structure

Anthropic and OpenAI filing for IPO in the same month creates real short-term tailwinds for API users. Both companies have strong incentives to drive usage before they go public, and that means lower prices and faster feature development.

But I'd call this market situation overblown if you treat the current pricing as permanent. Post-IPO shareholder pressure, gradual lock-in effects, and the data sovereignty issues around cheap Chinese models are all structural constraints that a price drop doesn't resolve.

The right stance is simple: take advantage of falling prices, but build in a way that keeps you mobile. The developer who maintains clean provider abstraction and optimizes aggressively for what's available now will come out ahead regardless of what the post-IPO pricing looks like.