BF16 vs FP8 vs INT4: The Quantization Bakeoff That Explains Why Your Local AI Agent Breaks

The tool call was perfect for the first 40 minutes. Your local agent, running a 27B model you quantized to fit your GPU, was reading a long log file, calling tools, editing files. Then it emitted malformed JSON. Then it called a tool that does not exist. Then it looped. You checked your Spring Boot wiring, your MCP server config, your prompt template. Everything was fine. So you blamed the model: "this 27B is just not smart enough for agentic work."

There is a decent chance the model was fine and your inference stack was not. A deeply technical experiment published on Level1Techs this week, currently sitting at 381 points on Hacker News with 142 comments, measured exactly this failure mode, and the numbers should change how every team running local models for agents picks their quantization (discussion here).

Full disclosure before anything else: the experiments below were run by the Level1Techs author on their hardware, not by me. I run local models behind my own Spring Boot services, and like most people I picked quantizations by asking "what fits in my VRAM?" This article is my attempt to translate their measurements into decisions you can actually apply, because the gap between what quantization marketing claims and what agent workloads experience is enormous.

What Was Actually Tested

The setup is more rigorous than most quantization claims you will read. The author took the official BF16 checkpoint of Qwen3.6-27B, a dense but hybrid model with 64 layers in a pattern of three linear-attention (Gated DeltaNet) layers followed by one full-attention layer, and ran it on an RTX PRO 6000 Blackwell GPU with a pinned nightly vLLM build. Eager execution, no CUDA graphs, no speculative decoding, no prefix caching. One variable changed at a time.

The workload matters even more than the hardware. Instead of a synthetic needle-in-a-haystack test, they replayed a roughly 100K-token context captured from a real agentic workstream, full of actual tool calls and real work products. As the author puts it, nobody could have benchmark-maxed for this prompt or calibrated their quant around it, because it does not exist in any public training or benchmark set.

The measurement method is the part worth understanding, because it explains why your agent breaks in ways a chat benchmark never shows. At every 32nd prompt token, they captured the full-vocabulary logits in BF16 and compared probability distributions in FP64 afterward. The headline metric is "top-1 flips": positions where a given configuration would have chosen a different greedy next token than the baseline. Every configuration was evaluated against the same forced token history, so the comparison stays mathematically controlled. When 2% of next-token decisions flip, small divergences compound with every generated token until you are somewhere else entirely.

Finding 1: Even the Attention Backend Changes the Answer

The first test is the most unsettling one. Same weights, same GPU, same everything, except the full-attention backend vLLM selected during prefill: FlashAttention 2, FlashInfer, or Triton Attention. Only 16 of the 64 layers even use this selectable backend.

For the first several thousand tokens, all backends agreed on the next token. Deeper into the prompt, they started disagreeing, and the disagreements appeared in clusters that varied with prompt content rather than growing smoothly with context length. To rule out randomness, the same backend was run multiple times: the logits were bit-for-bit identical across runs. The divergence comes purely from different matrix multiplication implementations computing the same math differently.

Read that again. The same model file, on the same GPU, with only the attention kernel swapped, produces measurably different next-token choices in long contexts. The nightly vLLM container alone shipped with 734 Python packages, each with its own bugs and quirks. Your local stack is a specific path through that mountain of code, and it is not the same path the model's authors took when they published their benchmark numbers.

Finding 2: KV Cache Quantization Is Where Long Agents Die

The second test has a title that should be printed on every local-LLM tutorial: "why your LLM's IQ drops like a rock after 40k tokens." This time the weights stayed BF16 and only the KV cache was quantized, the cache of past keys and values that grows with every token of context.

The result: with an INT8 KV cache, enough top-tokens flipped during tool calls that the author could produce a completely reproducible tool-calling error. The BF16 baseline completed the task fine. The INT8 cache eventually managed to recover. The INT4 KV cache did not recover at all.

This is the single most actionable finding for anyone running agents. Chat applications rarely exceed a few thousand tokens of context, so a quantized KV cache that doubles your effective context length feels like free wins. Agentic workloads live at 40K, 80K, 100K tokens, exactly where the KV cache error accumulates, and the failure mode lands precisely on the structured output an agent depends on: tool call syntax.

If your local agent works in short sessions and falls apart in long ones, and you are running a quantized KV cache to save memory, you have likely found your bug.

Finding 3: The Five-Way Weight Quantization Bakeoff

The main event compares five versions of the same model:

• BF16 reference: the official Qwen/Qwen3.6-27B checkpoint, unquantized
• Official FP8: Qwen's own Qwen3.6-27B-FP8, E4M3 weights in 128x128 blocks with dynamic FP8 activations
• INT8 W8A16: a community quant by TheHouseOfTheDude, static symmetric INT8 weights with BF16 activations, no calibration dataset
• NVIDIA NVFP4: nvidia's official Qwen3.6-27B-NVFP4, a mixed checkpoint of FP8 and 4-bit targets
• AWQ INT4 W4A16: a community AWQ quant, 4-bit weights with group size 32, calibrated on a disclosed "STEM and Agentic" dataset

The KV cache stayed BF16 for all five, isolating weight precision. The outcome, in the author's words, "shakes out fairly predictably" once you see it, but predictably is not the same as what the marketing suggests:

• The community INT8 quant beat everyone, including the first-party FP8 release from the model's own creators. The author attributes its fidelity partly to W8A16 (activations stay BF16) and partly to leaving the GDN projections unquantized.
• NVIDIA's NVFP4 release came in dead last, hitting roughly 50% token flips by 88K context. Half of the greedy next-token decisions differed from the reference.
• The failures were not cosmetic. Both 4-bit options (NVFP4 and AWQ INT4) failed to properly close their tool calls and botched a Cisco CLI syntax check, executing show run when the correct command was show arp. Both FP8 and INT8 completed the correct calls.

That last bullet is the whole story for agent builders. A 50% token-flip rate does not mean the model answers 50% of trivia questions wrong. It means that in long agentic contexts, the model you are running is measurably not the model you downloaded, and the divergence concentrates exactly where agents are most brittle: structured tool output.

There is a caveat worth keeping honest: on this particular run, vLLM classified the GPU path as lacking native FP4 support and fell back to weight-only FP4 compression through Marlin kernels. So this specific NVFP4 number reflects that runtime path, not necessarily native FP4 hardware arithmetic everywhere. The broader lesson survives the caveat: quantization claims are meaningless without the full runtime context behind them.

Why This Keeps Happening: KL Divergence Marketing

The post also explains why HF model cards get away with it. Quantized model cards often advertise impossibly low KL divergence from the reference model. The author's warning is direct: you cannot interpret that number unless the author discloses the reference checkpoints, the full runtime environment, the evaluation text, calibration data, context lengths, sampled positions, KL direction, vocabulary truncation, and aggregation method. Plenty of people get it wrong.

A KLD number measured on short chat prompts tells you nothing about behavior at 90K tokens of tool-call traffic. It is the inference-stack equivalent of a microbenchmark that never touches your actual workload.

What I Would Do Differently: A Checklist for Local Agent Stacks

Here is the decision list I would hand anyone running local models behind Spring AI, LangChain4j, or plain OpenAI-compatible clients:

• Default to W8A16 INT8 or first-party FP8 for agent workloads. In this bakeoff, both preserved tool-calling through 90K+ tokens. 4-bit weight quants are for chat, summarization, and batch classification, not for agents that must emit exact tool syntax after an hour of context.
• Keep the KV cache at BF16 for agents. The memory savings of a quantized KV cache are real, but test 2 shows the cost lands on tool calls in long sessions. If you must quantize the KV cache, INT8 recovered where INT4 did not; treat INT4 KV as disqualified for agents.
• Do not trust KLD numbers on model cards. If the methodology is not disclosed, the number is decoration.
• Test with your real workload, not prompts. Replay a captured 50K-100K token agentic session, tool calls included, and diff behavior against a short session. The author's core advice applies: zero-shot tests with three prompts are not an analog for agentic tasks.
• Pin your inference stack and treat changes as behavior changes. If bit-for-bit identical reruns can still diverge across attention backends, then a vLLM version bump is capable of silently changing your agent's behavior. Pin versions, and re-run your replay test after upgrades.
• Check sampler settings against the model card. A small bonus from the post: if your reasoning model loops endlessly inside its thinking output, your temperature is probably too low. The card specifies the intended settings; use them.

The uncomfortable summary: when your local agent gets "dumber" in long sessions, the model may be innocent. The quantization format, the KV cache precision, and the specific kernels your stack selected are all silently reshaping its decisions, and only a replay of your actual workload will show you by how much.

The Takeaway

Quantization is a trade between memory and fidelity, and the Level1Techs data gives that trade real numbers for the first time in a workload that resembles what local agents actually do. INT8 W8A16 community quants can beat first-party FP8. NVIDIA's 4-bit release flipped half its tokens at 88K context and broke tool calls. The KV cache quietly turned out to be the long-context killer. Any one of these can be the difference between "this model can't do agents" and "this model runs agents fine."

I write about Java, Spring Boot, and practical AI engineering every week. Subscribe, it's free.

Have you run a local model as an agent and hit mysterious tool-call failures in long sessions? What quantization are you running, and did this change how you think about it?