<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Susheem Koul</title>
    <description>The latest articles on DEV Community by Susheem Koul (@susheem-k).</description>
    <link>https://dev.to/susheem-k</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3400154%2Fb33cb2ca-ce29-449f-8da8-ca6c4aa84856.jpg</url>
      <title>DEV Community: Susheem Koul</title>
      <link>https://dev.to/susheem-k</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/susheem-k"/>
    <language>en</language>
    <item>
      <title>Training for quantization: QAT, 1-bit BitNet and 1.58-bit weights</title>
      <dc:creator>Susheem Koul</dc:creator>
      <pubDate>Sat, 17 Oct 2026 13:17:42 +0000</pubDate>
      <link>https://dev.to/susheem-k/training-for-quantization-qat-1-bit-bitnet-and-158-bit-weights-32b9</link>
      <guid>https://dev.to/susheem-k/training-for-quantization-qat-1-bit-bitnet-and-158-bit-weights-32b9</guid>
      <description>&lt;p&gt;In the &lt;a href="https://dev.to/susheem-k/how-8-bit-quantization-shrinks-an-llm-to-a-quarter-of-its-size-and-why-a-single-outlier-weight-3fa9"&gt;last post&lt;/a&gt; we quantized a model that was already trained (post-training quantization, PTQ). The weights were fixed, and we looked for the best way to squeeze them onto an integer grid.&lt;/p&gt;

&lt;p&gt;But what if the model &lt;em&gt;knew&lt;/em&gt; it was going to be quantized while it was learning? That is quantization-aware training (QAT). QAT is what eventually gives you BitNet, where every weight is -1, 0 or +1. Let's start with the basics of QAT.&lt;/p&gt;

&lt;h2&gt;
  
  
  Part 1: QAT
&lt;/h2&gt;

&lt;h3&gt;
  
  
  The idea
&lt;/h3&gt;

&lt;p&gt;PTQ: you train and retain the weights in full precision, then quantize them and deliver the quantized model for inference.&lt;/p&gt;

&lt;p&gt;QAT: you put the quantization of weights &lt;em&gt;inside&lt;/em&gt; the training loop. On every forward pass the weights are quantized, and the loss is computed with the quantized values. The optimizer then adjusts the weights so that the model does well &lt;em&gt;despite&lt;/em&gt; the rounding.&lt;/p&gt;

&lt;h3&gt;
  
  
  How does the training change to account for quantization?
&lt;/h3&gt;

&lt;p&gt;Normal training finds a point where the loss is low, and it stops caring about the shape of the loss around that point. What I mean by shape of the loss, is that if you change the weights from that minima, how quickly does the loss functions value rise again. That is fine with standard training practices, because once you publish the model the weights never move.&lt;/p&gt;

&lt;p&gt;Quantization breaks that assumption. Each weight now lands somewhere within a small error range δ of where training left it. This is the diff in weights that can come due to Quantization. So what matters is not just how low the loss is at the minimum, but &lt;strong&gt;how fast the loss climbs as you move away from it&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Near a minimum the slope is zero, so the loss increase from a nudge of size δ is approximately&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;ΔL ≈ ½ · H · δ²
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;where H is the curvature (second derivative, in simple terms how quickly the loss changes with a change in weight). Two minima with the same loss can behave very differently:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Sharp minimum&lt;/strong&gt; (large H): a small rounding error costs a lot of loss.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Wide minimum&lt;/strong&gt; (small H): the same rounding error barely matters.&lt;/li&gt;
&lt;/ul&gt;

&lt;blockquote&gt;
&lt;p&gt;A note on terms, because I kept mixing them up: &lt;em&gt;slope&lt;/em&gt; is the first derivative, and it is zero at both minima. &lt;em&gt;Curvature&lt;/em&gt; is the second derivative, and it is what differs.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9ozjkrro1afi11rv3bok.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9ozjkrro1afi11rv3bok.png" alt="Loss, slope and curvature for a sharp and a wide minimum, with a quantization error of 0.60 giving an 8x larger loss rise on the sharp one" width="800" height="812"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Try it yourself to build up your intuition for this, focus on how the loss changes with the change in weight from the point of minima for both wide and sharp minimas: &lt;a href="https://susheem-k.github.io/quant-viz/slope-curvature.html" rel="noopener noreferrer"&gt;slope and curvature playground&lt;/a&gt;. Drag the quantization error and watch the loss rise 8x faster on the sharp minimum.&lt;/p&gt;

&lt;p&gt;So QAT optimizes two things at once: a low loss, and a gentle curvature around it so quantization error stays in a controlled range.&lt;/p&gt;

&lt;h3&gt;
  
  
  How the gradient flows: fake quantization and the STE
&lt;/h3&gt;

&lt;p&gt;There is a catch. Rounding has a gradient of zero almost everywhere, so backprop would learn nothing through it.&lt;/p&gt;

&lt;p&gt;The fix is the &lt;strong&gt;straight-through estimator (STE)&lt;/strong&gt;: in the forward pass, use the quantized value. In the backward pass, pretend the rounding was the identity function and pass the gradient straight through.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;fake_quant&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;bits&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;qmax&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt; &lt;span class="o"&gt;**&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;bits&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="n"&gt;scale&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;qmax&lt;/span&gt;
    &lt;span class="n"&gt;w_q&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;torch&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;scale&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;clamp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;qmax&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;qmax&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;scale&lt;/span&gt;
    &lt;span class="c1"&gt;# forward: w_q. backward: gradient of w (identity)
&lt;/span&gt;    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w_q&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;detach&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The model computes with rounded weights, but the optimizer updates the underlying full-precision weights. Hold on to that idea, because BitNet pushes it to the limit.&lt;/p&gt;

&lt;p&gt;In practice QAT tends to beat PTQ, most visibly at low bit-widths such as 4-bit.&lt;/p&gt;

&lt;h2&gt;
  
  
  Part 2: BitNet and 1-bit weights
&lt;/h2&gt;

&lt;p&gt;Paper: &lt;a href="https://arxiv.org/abs/2310.11453" rel="noopener noreferrer"&gt;BitNet: Scaling 1-bit Transformers for Large Language Models&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  First, what is a linear layer?
&lt;/h3&gt;

&lt;p&gt;A linear layer is the most basic building block in a neural network: it takes a vector of numbers in, multiplies it by a matrix of learned numbers (the &lt;strong&gt;weights&lt;/strong&gt;), and gives a new vector out.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;output = input × weights   (+ a bias, which I'll ignore here)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Almost all of a model's parameters are entries in weight matrices like this. Two words I'll keep using:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Weights&lt;/strong&gt; are the stored, learned numbers. They are fixed once training is done.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Activations&lt;/strong&gt; are the numbers flowing &lt;em&gt;through&lt;/em&gt; the model for a particular input. They change with every token.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In a normal model, both are FP16.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where linear layers sit in a transformer
&lt;/h3&gt;

&lt;p&gt;A transformer is a stack of identical blocks. Each block has two parts, and almost everything with weights in it is a linear layer:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Attention&lt;/strong&gt; has four: the &lt;strong&gt;Q&lt;/strong&gt;, &lt;strong&gt;K&lt;/strong&gt; and &lt;strong&gt;V&lt;/strong&gt; projections, and an &lt;strong&gt;output projection&lt;/strong&gt;. The step in the middle, &lt;code&gt;softmax(QKᵀ)V&lt;/code&gt;, multiplies two sets of activations together. It has no stored weights, so it isn't a linear layer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The feed-forward network&lt;/strong&gt; has two: an &lt;strong&gt;up projection&lt;/strong&gt; and a &lt;strong&gt;down projection&lt;/strong&gt;, with a non-linearity (GELU) between them.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Watch &lt;a href="https://www.3blue1brown.com/lessons/mini-llm/" rel="noopener noreferrer"&gt;this &lt;/a&gt;if you want a quick crash course on Transformer Internals!&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9xvxtsjt6pj18tgkkul2.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9xvxtsjt6pj18tgkkul2.png" alt="A transformer block: attention with Q, K, V and output projections, and a feed-forward network with up and down projections, with the linear layers highlighted" width="800" height="480"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;That's why BitNet goes after the linear layers: shrink those, and you've shrunk the model.&lt;/p&gt;

&lt;h3&gt;
  
  
  So what does BitLinear change?
&lt;/h3&gt;

&lt;p&gt;BitNet replaces most of the &lt;code&gt;Linear&lt;/code&gt; with a &lt;strong&gt;BitLinear&lt;/strong&gt;. The job is the same (&lt;code&gt;output = input × weights&lt;/code&gt;), but the numbers going in are much smaller:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;Standard Linear&lt;/th&gt;
&lt;th&gt;BitLinear&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Weights&lt;/td&gt;
&lt;td&gt;FP16 (16 bits each)&lt;/td&gt;
&lt;td&gt;
&lt;strong&gt;±1&lt;/strong&gt; (1 bit each)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Activations&lt;/td&gt;
&lt;td&gt;FP16&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;INT8&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The matrix multiply&lt;/td&gt;
&lt;td&gt;FP16 multiply-adds&lt;/td&gt;
&lt;td&gt;INT8 inputs against ±1 weights&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Multiplying by +1 or -1 just keeps a number or flips its sign, so the matrix multiply collapses into &lt;strong&gt;additions and subtractions&lt;/strong&gt;. Multiplication is one of the most expensive things a chip does, so this saves time and energy as well as memory.&lt;/p&gt;

&lt;p&gt;Quantizing both sides loses scale information, though: a ±1 weight no longer says how big the original was. So BitLinear keeps two scale factors, one for the weights and one for the activations, and uses them to stretch the result back to the right size at the end. The recipe below shows how.&lt;/p&gt;

&lt;h3&gt;
  
  
  The BitLinear recipe
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Quantize the input&lt;/strong&gt; from FP16 to INT8 using absmax scaling (after a LayerNorm, so the activations have a steady scale to quantize).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Binarize the weights.&lt;/strong&gt; Take the mean of the weights, α. Anything above it becomes +1, anything below becomes -1. (Centring first keeps the +1s and -1s roughly balanced.)&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Matrix multiply&lt;/strong&gt; with the integer inputs and ±1 weights. This is mostly additions, which is much cheaper than multiplications.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Dequantize&lt;/strong&gt; the output using the activation scale and a weight scale β (the mean absolute weight), so the result lands back at the right magnitude.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;It is two quantizers layered together: one on the activations, one on the weights.&lt;/p&gt;

&lt;h3&gt;
  
  
  My question: why not just store 1 bit?
&lt;/h3&gt;

&lt;p&gt;If training is learning which weights should be "on" and which "off", why keep anything bigger than a bit?&lt;/p&gt;

&lt;p&gt;Because of the gradient. The binarizing function is &lt;code&gt;sign()&lt;/code&gt;. Its gradient is zero everywhere, except at zero where it is undefined. If the weights &lt;em&gt;were&lt;/em&gt; just bits, gradient descent would have nothing to update.&lt;/p&gt;

&lt;p&gt;So, as with QAT, the model keeps &lt;strong&gt;latent weights&lt;/strong&gt; in high precision (the paper trains with high-precision weights, FP16/BF16 in practice). The training loop looks like this:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Keep a full-precision latent weight.&lt;/li&gt;
&lt;li&gt;Forward pass: binarize it with &lt;code&gt;sign&lt;/code&gt;, use the ±1 version.&lt;/li&gt;
&lt;li&gt;Backward pass: use the STE, so the gradient lands on the latent weight.&lt;/li&gt;
&lt;li&gt;Update the latent weight.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;I first assumed the latent weights were INT8. That doesn't work, and it is a good illustration of why. Learning rates are small. A tiny gradient step on an INT8 weight would round back to the same integer, and the update would simply vanish. The latent weights need enough resolution to &lt;strong&gt;accumulate many small nudges&lt;/strong&gt; until one finally flips the sign.&lt;/p&gt;

&lt;p&gt;Only the ±1 version is needed at inference. The latent weights are a training-time scaffold.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fccjmhsn23xgxs3ctvecv.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fccjmhsn23xgxs3ctvecv.png" alt="After 10 training steps the 1-bit-only weight is stuck at -1, while the latent weight has crossed zero and flipped to +1" width="800" height="360"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Try it: &lt;a href="https://susheem-k.github.io/quant-viz/one-bit-weights.html" rel="noopener noreferrer"&gt;latent weights playground&lt;/a&gt;. Press "Train 10 steps": the 1-bit-only weight never moves, while the latent weight quietly accumulates the updates until it crosses zero and flips.&lt;/p&gt;

&lt;h3&gt;
  
  
  "So is 1-bit a sham?"
&lt;/h3&gt;

&lt;p&gt;I was suspicious of a claim that BitNet gets closer to FP16 quality as models grow. Doesn't that mean you just need a bigger model, and the savings vanish?&lt;/p&gt;

&lt;p&gt;No. The saving is &lt;strong&gt;per parameter&lt;/strong&gt;. A 1-bit model may need more parameters to match quality, but each parameter is up to 16x smaller than FP16. It only fails if you need more than 16x as many parameters. The trend in the paper is that the quality gap shrinks as the model scales.&lt;/p&gt;

&lt;h2&gt;
  
  
  Part 3: 1.58 bits
&lt;/h2&gt;

&lt;p&gt;A follow-up paper, &lt;a href="https://arxiv.org/abs/2402.17764" rel="noopener noreferrer"&gt;The Era of 1-bit LLMs (BitNet b1.58)&lt;/a&gt;, found that pure 1-bit needs a lot of scale to match full-precision quality. The authors added one more state: &lt;strong&gt;zero&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Weights are now -1, 0 or +1.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why zero helps
&lt;/h3&gt;

&lt;p&gt;A zero lets a weight say "this connection doesn't matter", instead of being forced to push the output up or down. The weight matrix becomes more expressive at almost no extra cost.&lt;/p&gt;

&lt;h3&gt;
  
  
  The recipe
&lt;/h3&gt;

&lt;ol&gt;
&lt;li&gt;Compute γ, the mean of the absolute weights.&lt;/li&gt;
&lt;li&gt;Divide each weight by γ and round to the nearest of -1, 0, +1.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Equivalently:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;|w| &amp;lt;= γ/2&lt;/code&gt; becomes &lt;strong&gt;0&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;w &amp;gt; γ/2&lt;/code&gt; becomes &lt;strong&gt;+1&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;w &amp;lt; -γ/2&lt;/code&gt; becomes &lt;strong&gt;-1&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;This is "absmean" quantization. The activations switch to symmetric absmax quantization (scaled around zero), rather than the 0-to-max range used before. Everything else about the training setup, including latent weights and the STE, stays the same.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;ternary_weights&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;eps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;1e-5&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;gamma&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;mean&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="nf"&gt;return &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;gamma&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="n"&gt;eps&lt;/span&gt;&lt;span class="p"&gt;)).&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;clamp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;gamma&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  The five-weight example
&lt;/h3&gt;

&lt;p&gt;Take &lt;code&gt;w = [0.9, -0.3, 0.05, -1.2, 0.4]&lt;/code&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;w0&lt;/th&gt;
&lt;th&gt;w1&lt;/th&gt;
&lt;th&gt;w2&lt;/th&gt;
&lt;th&gt;w3&lt;/th&gt;
&lt;th&gt;w4&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;original&lt;/td&gt;
&lt;td&gt;0.90&lt;/td&gt;
&lt;td&gt;-0.30&lt;/td&gt;
&lt;td&gt;0.05&lt;/td&gt;
&lt;td&gt;-1.20&lt;/td&gt;
&lt;td&gt;0.40&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;1-bit (β = 0.57)&lt;/td&gt;
&lt;td&gt;+0.57&lt;/td&gt;
&lt;td&gt;-0.57&lt;/td&gt;
&lt;td&gt;+0.57&lt;/td&gt;
&lt;td&gt;-0.57&lt;/td&gt;
&lt;td&gt;+0.57&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;ternary (γ = 0.57)&lt;/td&gt;
&lt;td&gt;+0.57&lt;/td&gt;
&lt;td&gt;-0.57&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;-0.57&lt;/td&gt;
&lt;td&gt;+0.57&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The tiny weight 0.05 is forced to ±0.57 in the 1-bit version, an error of 0.52. With a zero available, its error drops to 0.05. The other four are unchanged.&lt;/p&gt;

&lt;h3&gt;
  
  
  A bit of etymology : Why "1.58" bits?
&lt;/h3&gt;

&lt;p&gt;Three states need &lt;code&gt;log2(3) ≈ 1.58&lt;/code&gt; bits of information. But hardware cannot store 1.58 bits, so real implementations &lt;strong&gt;pack&lt;/strong&gt; the values. The simplest packing uses 2 bits per weight. A tighter one fits 5 ternary values into one byte (3⁵ = 243 ≤ 256), which is 1.6 bits per weight. So 1.58 is the information-theoretic figure, and practical storage lands slightly above it.&lt;/p&gt;

&lt;p&gt;Versus FP16, that is roughly 10x smaller per parameter.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to take away
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;PTQ&lt;/strong&gt; rounds a finished model. &lt;strong&gt;QAT&lt;/strong&gt; trains with the rounding in the loop, so the model finds weights that survive it.&lt;/li&gt;
&lt;li&gt;The reason it works is geometry: prefer &lt;strong&gt;wide minima&lt;/strong&gt; (low curvature) so quantization error costs little.&lt;/li&gt;
&lt;li&gt;Rounding has no gradient, so we use the &lt;strong&gt;STE&lt;/strong&gt; and keep &lt;strong&gt;high-precision latent weights&lt;/strong&gt; that accumulate small updates.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;BitNet&lt;/strong&gt; applies this with &lt;code&gt;sign()&lt;/code&gt;; &lt;strong&gt;b1.58&lt;/strong&gt; adds a zero, using absmean scaling, for a better quality/size trade-off.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;em&gt;Interactive pages: &lt;a href="https://susheem-k.github.io/quant-viz/slope-curvature.html" rel="noopener noreferrer"&gt;slope and curvature&lt;/a&gt;, &lt;a href="https://susheem-k.github.io/quant-viz/one-bit-weights.html" rel="noopener noreferrer"&gt;latent weights&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>machinelearning</category>
      <category>llm</category>
      <category>ai</category>
      <category>deeplearning</category>
    </item>
    <item>
      <title>How 8-bit quantization shrinks an LLM to half of its size — and why a single outlier weight can quietly ruin it</title>
      <dc:creator>Susheem Koul</dc:creator>
      <pubDate>Sun, 11 Oct 2026 07:50:16 +0000</pubDate>
      <link>https://dev.to/susheem-k/how-8-bit-quantization-shrinks-an-llm-to-a-quarter-of-its-size-and-why-a-single-outlier-weight-3fa9</link>
      <guid>https://dev.to/susheem-k/how-8-bit-quantization-shrinks-an-llm-to-a-quarter-of-its-size-and-why-a-single-outlier-weight-3fa9</guid>
      <description>&lt;p&gt;I've been reading about LLM quantization for a while, and almost every explanation I found stopped at the same sentence: "it makes the model smaller."&lt;/p&gt;

&lt;p&gt;Fine. But &lt;em&gt;how&lt;/em&gt;? And what do you give up?&lt;/p&gt;

&lt;p&gt;So I wrote my notes down, got stuck on a few things, and built a small playground to help me visualize this concept better.&lt;br&gt;
For this article, we only go as far as &lt;strong&gt;8-bit&lt;/strong&gt; quantization. The denser things like GPTQ, GGUF and BitNet are something I'm still reading and will cover in the next few weeks.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Want to poke at the playground first?&lt;/strong&gt; Here's the &lt;a href="https://susheem-k.github.io/quantization-playground/" rel="noopener noreferrer"&gt;link&lt;/a&gt; (&lt;a href="https://github.com/susheem-k/susheem-k.github.io/tree/main/quantization-playground" rel="noopener noreferrer"&gt;source code&lt;/a&gt;). It runs in your browser, no install. Here is a peek of how it looks like&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fr9kxeapdeszw19ft42rl.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fr9kxeapdeszw19ft42rl.png" alt="An image from the INT8 Quantization Playground showcasing block-wise quantization" width="800" height="921"&gt;&lt;/a&gt;&lt;/p&gt;
&lt;h2&gt;
  
  
  Why Quantize?
&lt;/h2&gt;

&lt;p&gt;Language models keep scaling. We have started hitting the Trillion mark in parameters. These weights have to live somewhere.&lt;/p&gt;

&lt;p&gt;A &lt;strong&gt;weight&lt;/strong&gt; is just one number the model learned during training. A model is billions, sometimes trillions, of them. The size of the overall weights is simple arithmetic: &lt;code&gt;number of weights × bytes per weight&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;Model weights are usually stored as floating-point numbers: &lt;strong&gt;Float32&lt;/strong&gt; (4 bytes each) or the 16-bit formats &lt;strong&gt;Float16&lt;/strong&gt; and &lt;strong&gt;BFloat16&lt;/strong&gt; (2 bytes each). These larger data types allow for a massive range as well as fine precision, which lets the models represent a wide variety of information. &lt;strong&gt;INT8&lt;/strong&gt; on the other hand is a plain 8-bit integer: 1 byte, and only 256 possible values, from -128 to 127.&lt;/p&gt;

&lt;p&gt;Here's the size math for an 8-billion-parameter model:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Format&lt;/th&gt;
&lt;th&gt;Bytes per weight&lt;/th&gt;
&lt;th&gt;Weights alone&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Float32&lt;/td&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;32 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Float16 / BFloat16&lt;/td&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;16 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;INT8&lt;/td&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;8 GB&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Quantization&lt;/strong&gt;, as a concept, is simply trying to store those weights in a smaller type, like INT8. The expectation is that you get a leaner, often faster, cheaper model, at the cost of some accuracy.&lt;/p&gt;

&lt;p&gt;Couple of things to know here though,&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Fitting the weights isn't the same as fitting the model.&lt;/strong&gt; 8 GB of INT8 weights already fills an 8 GB GPU. The model needs extra memory while it runs.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Why does a smaller model speed up inference?&lt;/strong&gt; There are two reasons.

&lt;ul&gt;
&lt;li&gt;If the smaller model now fits entirely in the GPU where the bigger one didn't, you no longer have to play fetch, pulling weights into the GPU from the much slower CPU memory.&lt;/li&gt;
&lt;li&gt;Even when both versions fit, the smaller one is still faster. To generate every single token, the GPU has to read the weights from its own memory into the cores that do the math. That reading is usually the bottleneck, not the math itself. Fewer bytes per weight means less time spent reading.&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;So the whole question is: how do you squeeze a float into one of 256 values without wrecking it?&lt;/p&gt;

&lt;p&gt;Here's the picture that made it click for me.&lt;/p&gt;

&lt;p&gt;Imagine a ruler with exactly &lt;strong&gt;256 evenly spaced marks&lt;/strong&gt;, from 0 to 255. I give you 1000 numbers, anywhere between 0 and 2550, so most of them are bigger than 255. You have a simple task: assign every one of the 1000 numbers to a mark on the ruler.&lt;/p&gt;

&lt;p&gt;One way to do it is to divide each number by 10 and put it on the nearest mark. A number like 510 lands on mark 51. The biggest possible number, 2550, lands on mark 255, the very last one.&lt;/p&gt;

&lt;p&gt;This is &lt;strong&gt;quantization&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Now if I give you back the ruler and ask you to convert mark 51 back into the original number, you simply multiply it by 10, and you have 510 again.&lt;/p&gt;

&lt;p&gt;This is &lt;strong&gt;dequantization&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;However, since there are far more numbers than marks, you will end up putting multiple numbers on the same mark. For example, 511 divided by 10 is 51.1, and since the ruler has nothing between 51 and 52, it also lands on mark 51. You keep a relative sense of magnitude, since bigger numbers still land on marks closer to 255. But the fact remains that you will &lt;strong&gt;squish&lt;/strong&gt; some of the numbers together into the same mark.&lt;/p&gt;

&lt;p&gt;So when you convert back, both 510 and 511 become 510. The 511 came back off by 1. That difference between the original number and what you get back after quantizing and dequantizing is the &lt;strong&gt;quantization error&lt;/strong&gt;. This degrades the model output quality.&lt;/p&gt;

&lt;p&gt;That 10 we divided by has a name by the way: the &lt;strong&gt;scale&lt;/strong&gt;, also called the &lt;strong&gt;step size&lt;/strong&gt;. It's the distance between two neighbouring marks, measured in the original numbers' units. And it wasn't random. It's the largest number divided by the last mark: 2550 / 255 = 10. That way the biggest number lands exactly on the end of the ruler.&lt;/p&gt;

&lt;p&gt;This is the higher level concept. The rest of this article is about doing it well.&lt;/p&gt;

&lt;p&gt;One last detail before we use real weights. INT8 numbers its 256 marks from -128 to 127 instead of 0 to 255. It's the same ruler, just shifted so that zero sits in the middle, which suits weights because they can be negative.&lt;/p&gt;
&lt;h2&gt;
  
  
  The math behind quantization
&lt;/h2&gt;

&lt;p&gt;Let's use five weights and follow them through the whole post:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mf"&gt;1.213&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.301&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.309&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;0.907&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;2.54&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The simplest choice is to centre the ruler on zero and make the largest weight land exactly on the last mark, 127 on the INT8 scale. This is &lt;strong&gt;symmetric quantization&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;The largest weight is 2.54, so:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;scale = 2.54 / 127 = 0.02
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every mark is now 0.02 apart. To &lt;strong&gt;quantize&lt;/strong&gt; a weight, divide it by the scale and round to the nearest whole number. To get a usable number back later, multiply by the scale again. That step is called &lt;strong&gt;dequantizing&lt;/strong&gt;.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Weight&lt;/th&gt;
&lt;th&gt;÷ 0.02&lt;/th&gt;
&lt;th&gt;Rounded (stored as INT8)&lt;/th&gt;
&lt;th&gt;× 0.02 (dequantized)&lt;/th&gt;
&lt;th&gt;Error&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;-1.213&lt;/td&gt;
&lt;td&gt;-60.65&lt;/td&gt;
&lt;td&gt;-61&lt;/td&gt;
&lt;td&gt;-1.22&lt;/td&gt;
&lt;td&gt;0.007&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.301&lt;/td&gt;
&lt;td&gt;15.05&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;td&gt;0.30&lt;/td&gt;
&lt;td&gt;0.001&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.309&lt;/td&gt;
&lt;td&gt;15.45&lt;/td&gt;
&lt;td&gt;15&lt;/td&gt;
&lt;td&gt;0.30&lt;/td&gt;
&lt;td&gt;0.009&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.907&lt;/td&gt;
&lt;td&gt;45.35&lt;/td&gt;
&lt;td&gt;45&lt;/td&gt;
&lt;td&gt;0.90&lt;/td&gt;
&lt;td&gt;0.007&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2.54&lt;/td&gt;
&lt;td&gt;127&lt;/td&gt;
&lt;td&gt;127&lt;/td&gt;
&lt;td&gt;2.54&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two things to notice.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Nothing comes back exactly.&lt;/strong&gt; The gap between the original and the dequantized value is the &lt;strong&gt;quantization error&lt;/strong&gt;. Because you round to the nearest mark, it's never more than half a step. Here that's 0.01.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;0.301 and 0.309 became the same number.&lt;/strong&gt; They were different weights, and now they're both 0.30. Any difference smaller than a step gets mushed into one mark. That's the information you lose.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flemq5v8vz17cl632bvuu.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flemq5v8vz17cl632bvuu.png" alt="Value Collapse" width="799" height="536"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;In the playground: 100 random weights between -1 and 1. Coral dots are weights that ended up sharing a mark with at least one other weight. (Settings: range -1 to 1, Gaussian, symmetric.)&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;In NumPy, the whole thing is a few lines:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;quantize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;scale&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;127&lt;/span&gt;
    &lt;span class="n"&gt;q&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;clip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;scale&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;127&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;127&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;astype&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;int8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;scale&lt;/span&gt;          &lt;span class="c1"&gt;# store these two
&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;dequantize&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;scale&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;astype&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;float32&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="n"&gt;scale&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;You store the INT8 values plus one scale. That's how 4 bytes per weight becomes 1.&lt;/p&gt;

&lt;h2&gt;
  
  
  Asymmetric Quantization
&lt;/h2&gt;

&lt;p&gt;Symmetric quantization forces us to keep the zero in the middle of the ruler. That's wasteful when your weights aren't spread evenly around zero.&lt;/p&gt;

&lt;p&gt;Say your values run from 0 to 1000. With symmetric quantization, 1000 lands on 127, so each step is 1000 / 127 ≈ 7.9. But nothing is negative, so the marks from -128 to -1 are never used. You've thrown away half the ruler.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Asymmetric quantization&lt;/strong&gt; fixes this. Instead of centring the ruler on zero, you slide it so its first mark sits on your smallest value and its last mark on your largest. Now all 256 marks cover real data, and each step is 1000 / 255 ≈ 3.9. That's twice as granular so you get better resolution and less &lt;strong&gt;squishing&lt;/strong&gt; of numbers.&lt;/p&gt;

&lt;p&gt;(It's 255 and not 256 because 256 marks have 255 gaps between them.)&lt;/p&gt;

&lt;p&gt;The catch: the float value 0.0 no longer lands on the INT8 value 0. So you store one more number, the &lt;strong&gt;zero-point&lt;/strong&gt;, which records which mark stands for 0.0. In the 0-to-1000 example, 0.0 sits on mark -128.&lt;/p&gt;

&lt;p&gt;So why would anyone pick symmetric? Because it's &lt;strong&gt;simpler and cheaper to compute&lt;/strong&gt;. With no zero-point to correct for, the integer math is cleaner. Weights are usually spread fairly evenly around zero anyway, so symmetric is a common default for weights.&lt;/p&gt;

&lt;h2&gt;
  
  
  The problem of outliers
&lt;/h2&gt;

&lt;p&gt;Let's add one more weight to our five, an &lt;strong&gt;outlier&lt;/strong&gt;: a value far bigger than the rest. It's 25.4, ten times our previous largest.&lt;/p&gt;

&lt;p&gt;The ruler has to stretch to reach it. The scale goes from 0.02 to 25.4 / 127 = 0.2. Every mark is now ten times further apart.&lt;/p&gt;

&lt;p&gt;Here's what happens to our original five:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Weight&lt;/th&gt;
&lt;th&gt;Before the outlier&lt;/th&gt;
&lt;th&gt;After the outlier&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;-1.213&lt;/td&gt;
&lt;td&gt;-1.22&lt;/td&gt;
&lt;td&gt;-1.20&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.301&lt;/td&gt;
&lt;td&gt;0.30&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.40&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.309&lt;/td&gt;
&lt;td&gt;0.30&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0.40&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;0.907&lt;/td&gt;
&lt;td&gt;0.90&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;1.00&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2.54&lt;/td&gt;
&lt;td&gt;2.54&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;2.60&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Errors that were under 0.01 are now up to 0.1. The weight 0.301 now comes back as 0.40, off by a third.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One big weight made everyone else ten times worse.&lt;/strong&gt; It didn't even have to be wrong. It just had to be big, because the largest value sets the scale for everything sharing the ruler.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl7gexxsfpfym0ib2gnbz.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fl7gexxsfpfym0ib2gnbz.png" alt="Outlier Squish" width="800" height="629"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;99 weights packed tightly around 0, plus one outlier at 1.0. The outlier stretches the ruler, so the whole cluster collapses onto a handful of marks: 22 different weights all become 0.00. (Settings: range -1 to 1, tight cluster + one outlier, symmetric.)&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;This isn't just my toy example. I tried the same thing on 4,096 random weights. Changing a single one to a large value made the error on the other 4,095 about 12.8× worse.&lt;/p&gt;

&lt;p&gt;So the real question in quantization isn't "how do I round?" It's "what do I do about outliers?"&lt;/p&gt;

&lt;h2&gt;
  
  
  Two fixes: clip it, or give it its own ruler
&lt;/h2&gt;

&lt;h3&gt;
  
  
  Fix 1: clipping
&lt;/h3&gt;

&lt;p&gt;Don't stretch the ruler to reach the outlier. Keep it where it was and let the outlier get &lt;strong&gt;clipped&lt;/strong&gt;, which means it's clamped to the last mark.&lt;/p&gt;

&lt;p&gt;In our example, that means keeping the scale at 0.02. Our five weights are back to their tiny errors.&lt;/p&gt;

&lt;p&gt;The price: the outlier comes back as 2.54 instead of 25.4. If that weight mattered to the model, you just damaged it.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9tlmdfu87othg7eti9yb.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F9tlmdfu87othg7eti9yb.png" alt="Clipping" width="800" height="572"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Same data with clipping on, zoomed in on the cluster. The step size drops from 0.00787 to 0.000156, so most weights get their own mark again (green). Notice the mean error actually goes up: the outlier went from 1.0 to about 0.02, and that one big error outweighs everything else. (Settings: clipping, keep central 98% of points.)&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;How far should you clip? There's no universal answer. You try a few ranges and measure how much error each one causes. Running sample data through the model to pick these ranges is called &lt;strong&gt;calibration&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Fix 2: more rulers
&lt;/h3&gt;

&lt;p&gt;Instead of one ruler for the whole weight matrix, split the weights into smaller &lt;strong&gt;groups&lt;/strong&gt; and give each group its own scale. An outlier can then only stretch the ruler for its own group.&lt;/p&gt;

&lt;p&gt;I like to think of it as limiting the blast radius.&lt;/p&gt;

&lt;p&gt;Split our six weights into two groups of three:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Group 1:&lt;/strong&gt; &lt;code&gt;[-1.213, 0.301, 0.309]&lt;/code&gt;. Its largest value is 1.213, so its scale is even finer than before, and its errors drop below 0.005.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Group 2:&lt;/strong&gt; &lt;code&gt;[0.907, 2.54, 25.4]&lt;/code&gt;. This group has the outlier, so its scale stretches to 0.2. Only these two neighbours take the hit.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Groups can be a whole row of the weight matrix (&lt;strong&gt;per-row&lt;/strong&gt;), a column (&lt;strong&gt;per-column&lt;/strong&gt;), or a short run of consecutive weights (&lt;strong&gt;block-wise&lt;/strong&gt;). Using one scale for everything is called &lt;strong&gt;per-tensor&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fuqfa0x510cdq5313bifi.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fuqfa0x510cdq5313bifi.png" alt="Per Tensor" width="800" height="515"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A 16 × 32 weight matrix with one outlier, quantized with a single scale for the whole matrix (per-tensor). Brighter means bigger error. One outlier, and the whole matrix lights up.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fith58vah61sfziv9ms2e.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fith58vah61sfziv9ms2e.png" alt="Block of 8" width="800" height="515"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Same matrix, one scale per block of 8 weights. Only the block holding the outlier lights up. Mean error drops from 0.0339 to 0.00341, about 10×, for roughly 2 extra bits per weight.&lt;/em&gt;&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;quantize_blockwise&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reshape&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;                         &lt;span class="c1"&gt;# one row per block
&lt;/span&gt;    &lt;span class="n"&gt;scale&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;abs&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;axis&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;keepdims&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mi"&gt;127&lt;/span&gt;
    &lt;span class="n"&gt;scale&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;where&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;scale&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;scale&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;         &lt;span class="c1"&gt;# avoid dividing by zero
&lt;/span&gt;    &lt;span class="n"&gt;q&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;clip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="n"&gt;scale&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;127&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;127&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;astype&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;int8&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;scale&lt;/span&gt;                                  &lt;span class="c1"&gt;# one scale per block
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Nothing is free, though. Every group needs its own scale stored next to it. With blocks of 64 and a 16-bit scale per block, that's an extra quarter of a bit per weight. More groups means a smaller blast radius, but more to store.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it yourself
&lt;/h2&gt;

&lt;p&gt;I got tired of imagining this, so I built the &lt;a href="https://susheem-k.github.io/quantization-playground/" rel="noopener noreferrer"&gt;playground&lt;/a&gt;. &lt;/p&gt;

&lt;p&gt;Here's what I'd try first:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Add a single outlier, then compare per-tensor with blocks of 8.&lt;/strong&gt; The average error drops roughly 10×.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Now look at the max error.&lt;/strong&gt; It barely moves. The weights sharing a block with the outlier still suffer. Smaller groups reduce &lt;em&gt;how many&lt;/em&gt; weights get hurt, not &lt;em&gt;how badly&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Make a whole column of outliers.&lt;/strong&gt; Per-column groups fix it, and per-row groups barely help. Switch to a row of outliers and it flips. Where the outliers live decides which grouping works.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fuch622amnwhzmviwy9pr.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fuch622amnwhzmviwy9pr.png" alt="Outlier Column" width="800" height="304"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;A whole column of outliers. Per-column scales win by a wide margin. Per-row barely helps, because every row contains an outlier.&lt;/em&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  So what all do we quantize in a model?
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Weight-only quantization.&lt;/strong&gt; The weights are stored in INT8, and right before each matrix multiply they're dequantized back to 16-bit floats. The math itself still runs in floating point. You save memory and the bandwidth spent reading weights, which is where most of the speedup comes from.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Weights and activations both in INT8&lt;/strong&gt; (often written &lt;strong&gt;W8A8&lt;/strong&gt;). First, a definition: &lt;strong&gt;activations&lt;/strong&gt; are the intermediate numbers flowing through the model, the outputs of one layer that become the inputs to the next. If both the weights and the activations are INT8, the multiply itself can run on fast integer hardware, and the result is dequantized afterwards.&lt;/p&gt;

&lt;p&gt;My question was: why bother quantizing activations, if they're thrown away after each layer? Two reasons:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Integer math hardware needs &lt;em&gt;both&lt;/em&gt; inputs to be integers.&lt;/li&gt;
&lt;li&gt;Some activations aren't thrown away. The model keeps the attention keys and values for every token so far, in what's called the &lt;strong&gt;KV cache&lt;/strong&gt;, and in long conversations that takes a lot of memory.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;Activations are harder than weights, though. Weights are fixed after training, so you can study them once. Activations change with every input, so you can't know their range in advance.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this goes next
&lt;/h2&gt;

&lt;p&gt;That difficulty in quantizing activations is something I am exploring next. The &lt;a href="https://arxiv.org/abs/2208.07339" rel="noopener noreferrer"&gt;LLM.int8() paper&lt;/a&gt; found that in models from around 6.7B parameters up, a few dimensions of the activations carry huge values, consistently across layers. Naive 8-bit quantization breaks on them.&lt;/p&gt;

&lt;p&gt;Here's what's coming in the series:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;4-bit Quantization&lt;/strong&gt; Only 16 marks on the ruler, so plain rounding stops being good enough. That's where GPTQ comes in.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;BitNet.&lt;/strong&gt; The extreme end, with weights pushed down to just three values: -1, 0 and 1.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you want to go deeper before then, start with the sources:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization" rel="noopener noreferrer"&gt;A Visual Guide to Quantization&lt;/a&gt; by Maarten Grootendorst, which is what got me started&lt;/li&gt;
&lt;li&gt;&lt;a href="https://arxiv.org/abs/2208.07339" rel="noopener noreferrer"&gt;LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>python</category>
    </item>
    <item>
      <title>TheAgentPlane is inviting contributions for Hacktoberfest!

Help us build THE open source agent governance platform!</title>
      <dc:creator>Susheem Koul</dc:creator>
      <pubDate>Sat, 26 Sep 2026 15:51:38 +0000</pubDate>
      <link>https://dev.to/susheem-k/theagentplane-is-inviting-contributions-for-hacktoberfest-help-us-build-the-open-source-agent-3h7o</link>
      <guid>https://dev.to/susheem-k/theagentplane-is-inviting-contributions-for-hacktoberfest-help-us-build-the-open-source-agent-3h7o</guid>
      <description>&lt;div class="ltag__link--embedded"&gt;
  &lt;div class="crayons-story "&gt;
  &lt;a href="https://dev.to/tisha/hacktoberfest-2026-help-us-govern-and-reproduce-ai-agents-59n1" class="crayons-story__hidden-navigation-link"&gt;Hacktoberfest 2026: help us govern (and reproduce) AI agents&lt;/a&gt;


  &lt;div class="crayons-story__body crayons-story__body-full_post"&gt;
      &lt;a href="https://dev.to/tisha/hacktoberfest-2026-help-us-govern-and-reproduce-ai-agents-59n1" class="crayons-article__context-note crayons-article__context-note__feed"&gt;&lt;p&gt;Hacktoberfest: Maintainer Spotlight&lt;/p&gt;

&lt;/a&gt;
    &lt;div class="crayons-story__top"&gt;
      &lt;div class="crayons-story__meta"&gt;
        &lt;div class="crayons-story__author-pic"&gt;

          &lt;a href="/tisha" class="crayons-avatar  crayons-avatar--l  "&gt;
            &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3960087%2F461d1521-802a-4dcb-b11e-7f2a7d88b7e6.png" alt="tisha profile" class="crayons-avatar__image"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
        &lt;div&gt;
          &lt;div&gt;
            &lt;a href="/tisha" class="crayons-story__secondary fw-medium m:hidden"&gt;
              Tisha Chawla
            &lt;/a&gt;
            &lt;div class="profile-preview-card relative mb-4 s:mb-0 fw-medium hidden m:inline-block"&gt;
              
                Tisha Chawla
                
                
              
              &lt;div id="story-author-preview-content-4750358" class="profile-preview-card__content crayons-dropdown branded-7 p-4 pt-0"&gt;
                &lt;div class="gap-4 grid"&gt;
                  &lt;div class="-mt-4"&gt;
                    &lt;a href="/tisha" class="flex"&gt;
                      &lt;span class="crayons-avatar crayons-avatar--xl mr-2 shrink-0"&gt;
                        &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3960087%2F461d1521-802a-4dcb-b11e-7f2a7d88b7e6.png" class="crayons-avatar__image" alt=""&gt;
                      &lt;/span&gt;
                      &lt;span class="crayons-link crayons-subtitle-2 mt-5"&gt;Tisha Chawla&lt;/span&gt;
                    &lt;/a&gt;
                  &lt;/div&gt;
                  &lt;div class="print-hidden"&gt;
                    
                      Follow
                    
                  &lt;/div&gt;
                  &lt;div class="author-preview-metadata-container"&gt;&lt;/div&gt;
                &lt;/div&gt;
              &lt;/div&gt;
            &lt;/div&gt;

          &lt;/div&gt;
          &lt;a href="https://dev.to/tisha/hacktoberfest-2026-help-us-govern-and-reproduce-ai-agents-59n1" class="crayons-story__tertiary fs-xs"&gt;&lt;time&gt;Sep 26&lt;/time&gt;&lt;span class="time-ago-indicator-initial-placeholder"&gt;&lt;/span&gt;&lt;/a&gt;
        &lt;/div&gt;
      &lt;/div&gt;

    &lt;/div&gt;

    &lt;div class="crayons-story__indention"&gt;
      &lt;h2 class="crayons-story__title crayons-story__title-full_post"&gt;
        &lt;a href="https://dev.to/tisha/hacktoberfest-2026-help-us-govern-and-reproduce-ai-agents-59n1" id="article-link-4750358"&gt;
          Hacktoberfest 2026: help us govern (and reproduce) AI agents
        &lt;/a&gt;
      &lt;/h2&gt;
        &lt;div class="crayons-story__tags"&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/hacktoberfest"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;hacktoberfest&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/opensource"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;opensource&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/ai"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;ai&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/python"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;python&lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="crayons-story__bottom"&gt;
        &lt;div class="crayons-story__details"&gt;
          &lt;a href="https://dev.to/tisha/hacktoberfest-2026-help-us-govern-and-reproduce-ai-agents-59n1" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left"&gt;
            &lt;div class="multiple_reactions_aggregate"&gt;
              &lt;span class="multiple_reactions_icons_container"&gt;
                  &lt;span class="crayons_icon_container"&gt;
                    &lt;img src="https://assets.dev.to/assets/multi-unicorn-b44d6f8c23cdd00964192bedc38af3e82463978aa611b4365bd33a0f1f4f3e97.svg" width="18" height="18"&gt;
                  &lt;/span&gt;
                  &lt;span class="crayons_icon_container"&gt;
                    &lt;img src="https://assets.dev.to/assets/sparkle-heart-5f9bee3767e18deb1bb725290cb151c25234768a0e9a2bd39370c382d02920cf.svg" width="18" height="18"&gt;
                  &lt;/span&gt;
              &lt;/span&gt;
              &lt;span class="aggregate_reactions_counter"&gt;4&lt;span class="hidden s:inline"&gt;&amp;nbsp;reactions&lt;/span&gt;&lt;/span&gt;
            &lt;/div&gt;
          &lt;/a&gt;
            &lt;a href="https://dev.to/tisha/hacktoberfest-2026-help-us-govern-and-reproduce-ai-agents-59n1#comments" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left flex items-center"&gt;
              

              1&lt;span class="hidden s:inline"&gt;&amp;nbsp;comment&lt;/span&gt;
            &lt;/a&gt;
        &lt;/div&gt;
        &lt;div class="crayons-story__save"&gt;
          &lt;small class="crayons-story__tertiary fs-xs mr-2"&gt;
            2 min read
          &lt;/small&gt;
        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;


</description>
      <category>agents</category>
      <category>ai</category>
      <category>hacktoberfest</category>
      <category>opensource</category>
    </item>
    <item>
      <title>Gartner Research predicts about 40% of AI projects will be shelved by 2027. It places unruly costs as one of the top reasons. Can you confidently say your agentic applications are optimized for spend? If not, read this and become a part of the discussion.</title>
      <dc:creator>Susheem Koul</dc:creator>
      <pubDate>Sat, 05 Sep 2026 11:19:56 +0000</pubDate>
      <link>https://dev.to/susheem-k/gartner-research-predicts-about-40-of-ai-projects-will-be-shelved-by-2027-it-places-unruly-costs-5ael</link>
      <guid>https://dev.to/susheem-k/gartner-research-predicts-about-40-of-ai-projects-will-be-shelved-by-2027-it-places-unruly-costs-5ael</guid>
      <description>&lt;div class="ltag__link--embedded"&gt;
  &lt;div class="crayons-story "&gt;
  &lt;a href="https://dev.to/tisha/did-your-ai-agent-quietly-blow-all-your-money-5e9c" class="crayons-story__hidden-navigation-link"&gt;Did your AI agent quietly blow all your money?&lt;/a&gt;


  &lt;div class="crayons-story__body crayons-story__body-full_post"&gt;
    &lt;div class="crayons-story__top"&gt;
      &lt;div class="crayons-story__meta"&gt;
        &lt;div class="crayons-story__author-pic"&gt;

          &lt;a href="/tisha" class="crayons-avatar  crayons-avatar--l  "&gt;
            &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3960087%2F461d1521-802a-4dcb-b11e-7f2a7d88b7e6.png" alt="tisha profile" class="crayons-avatar__image" width="800" height="800"&gt;
          &lt;/a&gt;
        &lt;/div&gt;
        &lt;div&gt;
          &lt;div&gt;
            &lt;a href="/tisha" class="crayons-story__secondary fw-medium m:hidden"&gt;
              Tisha Chawla
            &lt;/a&gt;
            &lt;div class="profile-preview-card relative mb-4 s:mb-0 fw-medium hidden m:inline-block"&gt;
              
                Tisha Chawla
                
                
              
              &lt;div id="story-author-preview-content-4580832" class="profile-preview-card__content crayons-dropdown branded-7 p-4 pt-0"&gt;
                &lt;div class="gap-4 grid"&gt;
                  &lt;div class="-mt-4"&gt;
                    &lt;a href="/tisha" class="flex"&gt;
                      &lt;span class="crayons-avatar crayons-avatar--xl mr-2 shrink-0"&gt;
                        &lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F3960087%2F461d1521-802a-4dcb-b11e-7f2a7d88b7e6.png" class="crayons-avatar__image" alt="" width="800" height="800"&gt;
                      &lt;/span&gt;
                      &lt;span class="crayons-link crayons-subtitle-2 mt-5"&gt;Tisha Chawla&lt;/span&gt;
                    &lt;/a&gt;
                  &lt;/div&gt;
                  &lt;div class="print-hidden"&gt;
                    
                      Follow
                    
                  &lt;/div&gt;
                  &lt;div class="author-preview-metadata-container"&gt;&lt;/div&gt;
                &lt;/div&gt;
              &lt;/div&gt;
            &lt;/div&gt;

          &lt;/div&gt;
          &lt;a href="https://dev.to/tisha/did-your-ai-agent-quietly-blow-all-your-money-5e9c" class="crayons-story__tertiary fs-xs"&gt;&lt;time&gt;Sep 5&lt;/time&gt;&lt;span class="time-ago-indicator-initial-placeholder"&gt;&lt;/span&gt;&lt;/a&gt;
        &lt;/div&gt;
      &lt;/div&gt;

    &lt;/div&gt;

    &lt;div class="crayons-story__indention"&gt;
      &lt;h2 class="crayons-story__title crayons-story__title-full_post"&gt;
        &lt;a href="https://dev.to/tisha/did-your-ai-agent-quietly-blow-all-your-money-5e9c" id="article-link-4580832"&gt;
          Did your AI agent quietly blow all your money?
        &lt;/a&gt;
      &lt;/h2&gt;
        &lt;div class="crayons-story__tags"&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/opensource"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;opensource&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/python"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;python&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/ai"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;ai&lt;/a&gt;
            &lt;a class="crayons-tag  crayons-tag--monochrome " href="/t/llm"&gt;&lt;span class="crayons-tag__prefix"&gt;#&lt;/span&gt;llm&lt;/a&gt;
        &lt;/div&gt;
      &lt;div class="crayons-story__bottom"&gt;
        &lt;div class="crayons-story__details"&gt;
          &lt;a href="https://dev.to/tisha/did-your-ai-agent-quietly-blow-all-your-money-5e9c" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left"&gt;
            &lt;div class="multiple_reactions_aggregate"&gt;
              &lt;span class="multiple_reactions_icons_container"&gt;
                  &lt;span class="crayons_icon_container"&gt;
                    &lt;img src="https://assets.dev.to/assets/fire-f60e7a582391810302117f987b22a8ef04a2fe0df7e3258a5f49332df1cec71e.svg" width="24" height="24"&gt;
                  &lt;/span&gt;
                  &lt;span class="crayons_icon_container"&gt;
                    &lt;img src="https://assets.dev.to/assets/multi-unicorn-b44d6f8c23cdd00964192bedc38af3e82463978aa611b4365bd33a0f1f4f3e97.svg" width="24" height="24"&gt;
                  &lt;/span&gt;
                  &lt;span class="crayons_icon_container"&gt;
                    &lt;img src="https://assets.dev.to/assets/sparkle-heart-5f9bee3767e18deb1bb725290cb151c25234768a0e9a2bd39370c382d02920cf.svg" width="24" height="24"&gt;
                  &lt;/span&gt;
              &lt;/span&gt;
              &lt;span class="aggregate_reactions_counter"&gt;5&lt;span class="hidden s:inline"&gt;&amp;nbsp;reactions&lt;/span&gt;&lt;/span&gt;
            &lt;/div&gt;
          &lt;/a&gt;
            &lt;a href="https://dev.to/tisha/did-your-ai-agent-quietly-blow-all-your-money-5e9c#comments" class="crayons-btn crayons-btn--s crayons-btn--ghost crayons-btn--icon-left flex items-center"&gt;
              

              2&lt;span class="hidden s:inline"&gt;&amp;nbsp;comments&lt;/span&gt;
            &lt;/a&gt;
        &lt;/div&gt;
        &lt;div class="crayons-story__save"&gt;
          &lt;small class="crayons-story__tertiary fs-xs mr-2"&gt;
            3 min read
          &lt;/small&gt;
        &lt;/div&gt;
      &lt;/div&gt;
    &lt;/div&gt;
  &lt;/div&gt;
&lt;/div&gt;

&lt;/div&gt;


</description>
    </item>
    <item>
      <title>I built an RPG that teaches Claude Code by making you actually use it</title>
      <dc:creator>Susheem Koul</dc:creator>
      <pubDate>Wed, 26 Aug 2026 15:14:07 +0000</pubDate>
      <link>https://dev.to/susheem-k/i-built-an-rpg-that-teaches-claude-code-by-making-you-actually-use-it-mlg</link>
      <guid>https://dev.to/susheem-k/i-built-an-rpg-that-teaches-claude-code-by-making-you-actually-use-it-mlg</guid>
      <description>&lt;p&gt;Most tool documentation teaches by just telling you things. You read a page about&lt;br&gt;
&lt;code&gt;/model&lt;/code&gt;, or hooks, or subagents, nod along, and forget it by the time you'd&lt;br&gt;
actually need it. I wanted something closer to how people actually learn a CLI,&lt;br&gt;
by using it, with something checking whether you did the thing right or not.&lt;/p&gt;

&lt;p&gt;So I built &lt;strong&gt;claude-quest&lt;/strong&gt;, a text RPG that runs entirely inside a real&lt;br&gt;
&lt;code&gt;claude&lt;/code&gt; session and teaches the Claude Code CLI zero to hero.&lt;/p&gt;
&lt;h2&gt;
  
  
  the idea isn't new, I borrowed it
&lt;/h2&gt;

&lt;p&gt;This is basically &lt;a href="https://github.com/phyver/GameShell" rel="noopener noreferrer"&gt;GameShell&lt;/a&gt;'s philosophy&lt;br&gt;
applied to Claude Code. GameShell teaches Unix shell commands by dropping you&lt;br&gt;
into a real shell wrapped in a themed fake filesystem, and grading your&lt;br&gt;
progress by checking real shell/filesystem state instead of asking you to self&lt;br&gt;
report or answer a quiz question.&lt;/p&gt;

&lt;p&gt;&lt;code&gt;claude-quest&lt;/code&gt; does the same thing, except the "filesystem" is a real Claude&lt;br&gt;
Code environment. missions live in real sandbox directories, and progress is&lt;br&gt;
checked by inspecting what you actually did, config files you wrote, hooks&lt;br&gt;
that fired, tool calls that happened.&lt;/p&gt;
&lt;h2&gt;
  
  
  what it actually looks like
&lt;/h2&gt;

&lt;p&gt;There's no separate app, no fancy terminal UI, none of that. You say "let's&lt;br&gt;
play claude quest" inside a normal &lt;code&gt;claude&lt;/code&gt; session, and Claude itself becomes&lt;br&gt;
the game master. It reads real output from the engine and narrates on top of&lt;br&gt;
it, something like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="gt"&gt;&amp;gt; lets play claude quest&lt;/span&gt;

&lt;span class="gs"&gt;**The Gatehall**&lt;/span&gt;

You've just been let through the outer gate. The tower keeps no secrets
from those who bother to read the walls...

&lt;span class="gs"&gt;**Mission: First Contact**&lt;/span&gt; (Tier 1, Artifact)

Create a CLAUDE.md file in this sandbox recording how to run the
project's tests. Let me know when you're done and I'll check it.
&lt;span class="gt"&gt;
&amp;gt; done&lt;/span&gt;

CLAUDE.md records how to run the tests. Claude Code will know next time.
MISSION_STATUS: complete

&lt;span class="gs"&gt;**What you actually learned:**&lt;/span&gt; Claude Code reads a file named CLAUDE.md
in your project root automatically, at the start of every session...
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;that's it, that's the whole interface. it's just a chat, with real commands&lt;br&gt;
running underneath it and real files getting checked. nothing gets rendered,&lt;br&gt;
nothing gets animated, it's plain text the way &lt;code&gt;claude&lt;/code&gt; sessions already look.&lt;/p&gt;
&lt;h2&gt;
  
  
  missions are graded on real state, not vibes
&lt;/h2&gt;

&lt;p&gt;Every mission is one of four tiers, and the grading gets less mechanical the&lt;br&gt;
further you go:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Artifact&lt;/strong&gt;, write or edit a file (a &lt;code&gt;CLAUDE.md&lt;/code&gt;, a config) and the engine
checks the file itself.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Invocation&lt;/strong&gt;, actually run a &lt;code&gt;claude&lt;/code&gt; session and do the thing (switch
models, compact, check cost). the engine checks what landed afterward
(like &lt;code&gt;.claude/settings.json&lt;/code&gt;), never what you said in the session.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Mastery&lt;/strong&gt;, fix something that's conceptually broken (a skill whose
description doesn't say what its for, a subagent that won't fire), then
the engine runs a held-out test battery against your own &lt;code&gt;claude&lt;/code&gt; CLI.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Judgment&lt;/strong&gt;, write something graded on a rubric (a commit message, a
skill description) and an isolated judge call scores it per criteria.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;None of these ask you to type an answer into a prompt, they ask you to leave&lt;br&gt;
a trace and then the engine checks the trace.&lt;/p&gt;
&lt;h2&gt;
  
  
  try it
&lt;/h2&gt;

&lt;p&gt;Inside any &lt;code&gt;claude&lt;/code&gt; session:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;/plugin marketplace add susheem-k/claude-quest
/plugin &lt;span class="nb"&gt;install &lt;/span&gt;claude-quest@claude-quest
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then just say "let's play claude quest." Progress saves to &lt;code&gt;~/.claude-quest/&lt;/code&gt;,&lt;br&gt;
so it doesn't really matter which project your in when you pick it back up.&lt;/p&gt;

&lt;h2&gt;
  
  
  where it's at, not all the levels exist yet
&lt;/h2&gt;

&lt;p&gt;This is a playable alpha, not a finished game. The full loop already works&lt;br&gt;
(characters, saves, hints, grading for all four tiers) but the campaign&lt;br&gt;
doesn't cover everything Claude Code can do yet. Right now there's only a&lt;br&gt;
handful of example missions, one or two per tier, across five arcs. More&lt;br&gt;
missions are getting added.&lt;/p&gt;

&lt;p&gt;And since this is open source, that part doesn't have to be just me. If&lt;br&gt;
there's a Claude Code feature you wish more people knew how to use, a&lt;br&gt;
command, a hook, a workflow you had to learn the hard way, I'd genuinely like to turn it into a mission. The mission format is just a directory with a goal, a hint, and a &lt;code&gt;check.js&lt;/code&gt; (the &lt;a href="https://github.com/susheem-k/claude-quest/blob/main/docs/DESIGN.md" rel="noopener noreferrer"&gt;design doc&lt;/a&gt; has the exact file contract for it), so adding one doesn't require touching&lt;br&gt;
the engine at all.&lt;/p&gt;

&lt;p&gt;Open an &lt;a href="https://github.com/susheem-k/claude-quest/issues" rel="noopener noreferrer"&gt;issue&lt;/a&gt; with a&lt;br&gt;
mission idea, or just send a PR with one built already, both are welcome.&lt;/p&gt;

&lt;p&gt;if you try it and it teaches you something, a star on the repo genuinely&lt;br&gt;
helps, it's how other people find it.&lt;/p&gt;

&lt;p&gt;Repo: &lt;a href="https://github.com/susheem-k/claude-quest" rel="noopener noreferrer"&gt;https://github.com/susheem-k/claude-quest&lt;/a&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>claude</category>
      <category>opensource</category>
      <category>node</category>
    </item>
    <item>
      <title>From Validating Code to Evaluating Behavior: Rethinking Testing for AI Agents</title>
      <dc:creator>Susheem Koul</dc:creator>
      <pubDate>Mon, 17 Aug 2026 17:25:57 +0000</pubDate>
      <link>https://dev.to/susheem-k/from-validating-code-to-evaluating-behavior-rethinking-testing-for-ai-agents-315g</link>
      <guid>https://dev.to/susheem-k/from-validating-code-to-evaluating-behavior-rethinking-testing-for-ai-agents-315g</guid>
      <description>&lt;p&gt;Traditional software testing assumes that system behavior is governed by &lt;strong&gt;deterministic code paths and configurations&lt;/strong&gt;. Once an LLM enters the execution loop, that assumption weakens. The challenge shifts from verifying that code followed the expected path to evaluating whether a system behaved appropriately and achieved the desired outcome.&lt;/p&gt;

&lt;p&gt;That distinction fundamentally changes what we test, how we write assertions, and even what &lt;strong&gt;"correctness"&lt;/strong&gt; means. We are moving from verifying deterministic code to evaluating probabilistic system behavior.&lt;/p&gt;

&lt;h2&gt;
  
  
  Traditional Code Testing and Where It Starts Breaking Down
&lt;/h2&gt;

&lt;p&gt;In traditional software systems, behavior is encapsulated into methods, APIs, configuration, and external dependencies. Even when a system becomes extremely complex, the behavior is still ultimately determined by code.&lt;/p&gt;

&lt;p&gt;The output of a request flowing through a service can be largely thought of as a combination of:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Output ≈ Input × Configuration × Infrastructure × Dependencies × Code Version
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The resulting state space may be enormous, but it remains largely open to systematic testing against an enumerated set of cases. This is the foundation upon which software testing is built. Cover enough of the state space and you gain confidence that the system will behave correctly.&lt;/p&gt;

&lt;h3&gt;
  
  
  A Concrete Example
&lt;/h3&gt;

&lt;p&gt;Consider a workflow that reads configuration from a config store, calls a customer profile API, calls a pricing API, applies deterministic business rules, and produces a recommendation. Testing this system is relatively straightforward. We can:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Mock the APIs&lt;/li&gt;
&lt;li&gt;Control configuration state&lt;/li&gt;
&lt;li&gt;Enumerate edge cases&lt;/li&gt;
&lt;li&gt;Verify outputs&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Distributed systems introduce their own challenges—network latency, node failures, and partial outages—but established testing techniques such as &lt;strong&gt;chaos engineering&lt;/strong&gt; can provide confidence that the system behaves correctly under those conditions.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where Things Change with LLMs
&lt;/h3&gt;

&lt;p&gt;Now replace the deterministic business rules with an &lt;strong&gt;LLM-driven planner&lt;/strong&gt;. The APIs, tools, and data remain unchanged, but the workflow must now decide:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Which tools to call&lt;/li&gt;
&lt;li&gt;In what order&lt;/li&gt;
&lt;li&gt;Whether more information is needed&lt;/li&gt;
&lt;li&gt;How to handle ambiguity&lt;/li&gt;
&lt;li&gt;How to formulate the final response&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The upside is obvious: the system becomes dramatically more flexible and can handle scenarios that were never explicitly programmed for.&lt;/p&gt;

&lt;p&gt;The downside is that &lt;strong&gt;the testing surface area grows significantly&lt;/strong&gt;. The behavior driver is no longer only code; it is now a probabilistic reasoning system operating inside the application itself. Even temperature-0 inference does not always guarantee perfectly reproducible outputs in many production LLM deployments due to model updates, infrastructure differences, and implementation details.&lt;/p&gt;

&lt;p&gt;The result is a system whose behavior is &lt;strong&gt;harder to predict, harder to bound, and therefore harder to test&lt;/strong&gt;.&lt;/p&gt;

&lt;h4&gt;
  
  
  The Testing Challenge Shifts
&lt;/h4&gt;

&lt;p&gt;From:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Did the code execute exactly as expected?"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;To:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Did the system behave appropriately?"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;Once we accept that agent behavior cannot always be validated through deterministic assertions, the next question becomes: what exactly are we evaluating?&lt;/p&gt;




&lt;h2&gt;
  
  
  What Are We Actually Testing?
&lt;/h2&gt;

&lt;p&gt;Agent testing significantly expands what correctness means. Instead of asking whether a function returned the correct value, we ask whether the &lt;strong&gt;overall behavior was acceptable&lt;/strong&gt;. A response can be factually correct while being incomplete, violate instructions, or arrive at the right answer through an inefficient or risky sequence of actions.&lt;/p&gt;

&lt;p&gt;As a result, evaluating an agent often means evaluating it across &lt;strong&gt;multiple dimensions&lt;/strong&gt;:&lt;/p&gt;

&lt;h3&gt;
  
  
  Outcome Metrics
&lt;/h3&gt;

&lt;p&gt;These metrics focus on whether the agent successfully completed the task.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Response Completeness&lt;/strong&gt; — Did the agent fully complete the task, or only address part of the request?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Response Correctness&lt;/strong&gt; — Is the answer factually accurate, grounded in the available context, and compliant with instructions?&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Operational Metrics
&lt;/h3&gt;

&lt;p&gt;These metrics evaluate how efficiently the agent performed.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Runtime and Latency&lt;/strong&gt; — How quickly does the agent respond and complete its work?&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost Efficiency&lt;/strong&gt; — How many tokens, model calls, and external resources were required to complete the task?&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Reliability Metrics
&lt;/h3&gt;

&lt;p&gt;These metrics measure how the system behaves when things do not go according to plan.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Failure Handling&lt;/strong&gt; — Does the agent recover gracefully from tool failures, timeouts, malformed inputs, and missing data?&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Quality Metrics
&lt;/h3&gt;

&lt;p&gt;These metrics focus on the usefulness and presentation of the final output.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Response Quality&lt;/strong&gt; — Is the output clear, concise, well-structured, and appropriate for the audience?&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Governance Metrics
&lt;/h3&gt;

&lt;p&gt;These metrics ensure the system operates within acceptable boundaries.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Security and Safety&lt;/strong&gt; — Is the agent resilient to prompt injection, jailbreaks, data exfiltration attempts, and other adversarial inputs?&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  Behavioral Metrics
&lt;/h3&gt;

&lt;p&gt;These metrics evaluate how the agent arrived at its answer, not just the answer itself.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Trajectory Quality&lt;/strong&gt; — Did the agent follow a sensible path to reach its answer, or waste effort on unnecessary reasoning and tool calls?&lt;/li&gt;
&lt;/ul&gt;

&lt;h3&gt;
  
  
  The Paradigm Shift
&lt;/h3&gt;

&lt;p&gt;Taken together, these dimensions reveal an important shift:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;We are no longer evaluating whether a piece of code returned the expected value. We are evaluating whether an intelligent system behaved effectively across a range of often competing objectives.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  How Do We Measure These Metrics?
&lt;/h2&gt;

&lt;p&gt;Once we know what we want to evaluate, the next challenge is figuring out how to measure it.&lt;/p&gt;

&lt;p&gt;In traditional software testing, assertions are usually &lt;strong&gt;deterministic&lt;/strong&gt;. We know the expected output ahead of time and can compare actual behavior against it.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="nf"&gt;calculate_tax&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;18&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Many agent evaluation dimensions do not naturally translate to this style of "hard" testing.&lt;/p&gt;

&lt;p&gt;How do we write deterministic assertions for questions such as:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Was the answer complete?&lt;/li&gt;
&lt;li&gt;Was the response faithful to the provided context?&lt;/li&gt;
&lt;li&gt;Did the agent use retrieved information appropriately?&lt;/li&gt;
&lt;li&gt;Was the tone professional?&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are &lt;strong&gt;semantic questions&lt;/strong&gt; rather than computational ones.&lt;/p&gt;

&lt;p&gt;As a result, modern AI evaluation frameworks have largely converged on three broad approaches.&lt;/p&gt;

&lt;h3&gt;
  
  
  1. Mathematical Metrics
&lt;/h3&gt;

&lt;p&gt;Some dimensions can be measured using statistical techniques.&lt;/p&gt;

&lt;p&gt;For example, here's an excerpt from the &lt;strong&gt;RAGAS metrics documentation&lt;/strong&gt; on Answer Relevancy:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;The Answer Relevancy metric measures how relevant a response is to the user input. It ranges from 0 to 1, with higher scores indicating better alignment with the user input.&lt;/p&gt;

&lt;p&gt;An answer is considered relevant if it directly and appropriately addresses the original question. This metric focuses on how well the answer matches the intent of the question, without evaluating factual accuracy. It penalizes answers that are incomplete or include unnecessary details.&lt;/p&gt;

&lt;p&gt;This metric is calculated using the &lt;code&gt;user_input&lt;/code&gt; and the &lt;code&gt;response&lt;/code&gt; as follows:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Generate a set of artificial questions (default is 3) based on the response. These questions are designed to reflect the content of the response.&lt;/li&gt;
&lt;li&gt;Compute the cosine similarity between the embedding of the user input (E_u) and the embedding of each generated question (E_q).&lt;/li&gt;
&lt;li&gt;Take the average of these cosine similarity scores to get the Answer Relevancy.
&lt;/li&gt;
&lt;/ol&gt;
&lt;/blockquote&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Answer Relevancy = (1/N) Σ cosine_similarity(E_q, E_u)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;a href="https://docs.ragas.io/en/latest/concepts/metrics/index.html" rel="noopener noreferrer"&gt;You can find the entire catalog of metrics provided by RAGAS here&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  2. Specialised Models
&lt;/h3&gt;

&lt;p&gt;Some evaluation dimensions are measured using dedicated machine learning models trained for specific tasks.&lt;/p&gt;

&lt;p&gt;Examples include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Sentiment analysis&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;PII detection&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Bias detection&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;In these cases, the evaluator itself is a model that has been optimised for a particular classification or scoring task.&lt;/p&gt;

&lt;h3&gt;
  
  
  3. LLM-as-a-Judge with Pre-defined Rubrics
&lt;/h3&gt;

&lt;p&gt;One of the increasingly common approaches is using &lt;strong&gt;another LLM to evaluate the output&lt;/strong&gt; of the system under test.&lt;/p&gt;

&lt;p&gt;Rather than comparing outputs against exact expected values, an evaluator model is asked to assess whether a response satisfies a particular rubric.&lt;/p&gt;

&lt;p&gt;Frameworks such as &lt;strong&gt;DeepEval&lt;/strong&gt; and &lt;strong&gt;RAGAS&lt;/strong&gt; package these techniques into reusable metrics that engineers can apply directly within their test suites.&lt;/p&gt;

&lt;p&gt;For example, rather than implementing faithfulness scoring from scratch, a framework may expose a pre-built metric:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;deepeval.metrics&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;FaithfulnessMetric&lt;/span&gt;

&lt;span class="n"&gt;metric&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;FaithfulnessMetric&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The result is that engineers can focus on &lt;strong&gt;defining what good behavior looks like for their application&lt;/strong&gt; rather than building evaluation logic from first principles.&lt;/p&gt;

&lt;h3&gt;
  
  
  Building Metrics on Top of These Approaches
&lt;/h3&gt;

&lt;p&gt;This is an important shift. In traditional software, we primarily wrote assertions against &lt;strong&gt;outputs&lt;/strong&gt;. In agent systems, we increasingly write assertions against &lt;strong&gt;behaviours&lt;/strong&gt;, and those behaviours are often quantified through metrics rather than exact expected values.&lt;/p&gt;




&lt;h2&gt;
  
  
  Traditional Testing Layers Still Matter
&lt;/h2&gt;

&lt;p&gt;Most traditional testing techniques remain just as important. The difference is that they now validate different parts of the system. An agent is still built on top of code, APIs, databases, configuration, authentication systems, and external dependencies. These deterministic components should continue to be tested using the same techniques that have worked for decades.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Unit tests&lt;/strong&gt; validate building blocks like API clients, parsers, and retry logic.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Integration tests&lt;/strong&gt; verify tool contracts and connections to databases or vector stores.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Functional/E2E tests&lt;/strong&gt; validate outcomes, ensuring the agent achieves goal Z given input X and environment Y.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Chaos and Performance tests&lt;/strong&gt; ensure the system remains performant and recovers gracefully from tool outages or timeouts.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Traditional testing therefore does not disappear in agent systems. It continues to provide confidence in the deterministic layers of the stack.&lt;/p&gt;




&lt;h2&gt;
  
  
  Testing Methodologies for AI Agents
&lt;/h2&gt;

&lt;p&gt;Understanding what to measure is only half the problem. The next challenge is designing tests that produce meaningful and repeatable signals.&lt;/p&gt;

&lt;h3&gt;
  
  
  Define the Test Unit
&lt;/h3&gt;

&lt;p&gt;The first step is deciding what a test actually represents. For agents, a test is usually centered around a &lt;strong&gt;scenario or behavior&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;A typical test contains:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;A &lt;strong&gt;scenario definition&lt;/strong&gt; describing the behavior being evaluated&lt;/li&gt;
&lt;li&gt;An &lt;strong&gt;input prompt&lt;/strong&gt; that triggers the scenario&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;controlled environment&lt;/strong&gt; consisting of tool responses, configuration, and datasets&lt;/li&gt;
&lt;li&gt;A &lt;strong&gt;set of assertions&lt;/strong&gt; used to evaluate the outcome&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Control the Environment
&lt;/h3&gt;

&lt;p&gt;One of the easiest ways to create flaky agent tests is to allow the environment to change between runs.&lt;/p&gt;

&lt;p&gt;Tool responses, configuration state, datasets, and external dependencies should be controlled wherever possible. This is no different from traditional testing, but &lt;strong&gt;becomes even more important because the LLM itself already introduces variability&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Without environmental control, it becomes difficult to determine whether a failure was caused by the agent or by a changing dependency.&lt;/p&gt;

&lt;h3&gt;
  
  
  Hard vs Soft Assertions
&lt;/h3&gt;

&lt;p&gt;Some behaviours are &lt;strong&gt;non-negotiable&lt;/strong&gt; and should fail immediately when violated. Examples include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Invalid schemas&lt;/li&gt;
&lt;li&gt;Incorrect computed values&lt;/li&gt;
&lt;li&gt;Required tool usage&lt;/li&gt;
&lt;li&gt;Missing mandatory fields&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are &lt;strong&gt;hard assertions&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Other behaviours are &lt;strong&gt;inherently subjective&lt;/strong&gt; and are often better represented as scores or thresholds. Examples include:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Completeness&lt;/li&gt;
&lt;li&gt;Clarity&lt;/li&gt;
&lt;li&gt;Tone&lt;/li&gt;
&lt;li&gt;Faithfulness&lt;/li&gt;
&lt;li&gt;Trajectory quality&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These are &lt;strong&gt;soft assertions&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;In practice, &lt;strong&gt;most agent tests combine both&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Account for Variability
&lt;/h3&gt;

&lt;p&gt;Because agent behavior is probabilistic, a &lt;strong&gt;single test execution is rarely sufficient&lt;/strong&gt;. Instead of asking:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"Did the test pass?"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;We must ask:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;"How consistently does the agent pass?"&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;&lt;strong&gt;Reliability is measured through success rates across multiple runs&lt;/strong&gt; rather than binary outcomes.&lt;/p&gt;




&lt;h2&gt;
  
  
  Common Capabilities Across Agent Testing Frameworks
&lt;/h2&gt;

&lt;p&gt;After exploring several agent evaluation frameworks, I found that most of them converge on a similar set of capabilities.&lt;/p&gt;

&lt;p&gt;Whether you're looking at &lt;strong&gt;DeepEval&lt;/strong&gt;, &lt;strong&gt;RAGAS&lt;/strong&gt;, &lt;strong&gt;LangSmith&lt;/strong&gt;, &lt;strong&gt;Arize&lt;/strong&gt; or other emerging tools, the implementation details vary but the core ideas remain largely the same.&lt;/p&gt;

&lt;p&gt;Most frameworks focus on &lt;strong&gt;three major areas&lt;/strong&gt;:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Capturing and understanding agent behavior&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Evaluating that behavior using reusable metrics&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Persisting and re-running evaluations over time&lt;/strong&gt;&lt;/li&gt;
&lt;/ol&gt;

&lt;h3&gt;
  
  
  Tracing and Observability
&lt;/h3&gt;

&lt;p&gt;Most modern frameworks provide some form of &lt;strong&gt;tracing or instrumentation&lt;/strong&gt;. This enables understanding:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;The sequence of tool calls&lt;/li&gt;
&lt;li&gt;Retrieved documents&lt;/li&gt;
&lt;li&gt;Intermediate decisions&lt;/li&gt;
&lt;li&gt;Model interactions&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;These captured data points provide the &lt;strong&gt;raw signals&lt;/strong&gt; from which many evaluation metrics are derived.&lt;/p&gt;

&lt;p&gt;For example, DeepEval allows developers to instrument portions of an application and collect execution traces that can later be inspected or evaluated:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;deepeval.tracing&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;observe&lt;/span&gt;

&lt;span class="nd"&gt;@observe&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;retrieve_documents&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;query&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="bp"&gt;...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h3&gt;
  
  
  Metrics and Assertions
&lt;/h3&gt;

&lt;p&gt;Once behavior can be observed, the next challenge is evaluating it.&lt;/p&gt;

&lt;p&gt;Most frameworks provide a &lt;strong&gt;collection of reusable metrics&lt;/strong&gt; that abstract away the complexity of evaluation. Instead of building custom scoring logic for every use case, teams can &lt;strong&gt;compose existing metrics into test assertions&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;A typical evaluation for DeepEval, for example, may look like:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;deepeval.metrics&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;FaithfulnessMetric&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;deepeval.test_case&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;LLMTestCase&lt;/span&gt;

&lt;span class="n"&gt;metric&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;FaithfulnessMetric&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="n"&gt;test_case&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;LLMTestCase&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="nb"&gt;input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;What is Kubernetes?&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;actual_output&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;response&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;retrieval_context&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;context&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;measure&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;test_case&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;assert&lt;/span&gt; &lt;span class="n"&gt;metric&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;score&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;&lt;/span&gt; &lt;span class="mf"&gt;0.8&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The important idea here is not the specific metric itself, but the &lt;strong&gt;abstraction&lt;/strong&gt;. Engineers are no longer writing assertions directly against outputs. They are increasingly &lt;strong&gt;writing assertions against behavioural qualities&lt;/strong&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Test Datasets and Regression Testing
&lt;/h3&gt;

&lt;p&gt;The final pillar is &lt;strong&gt;persistence&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Once tests have been defined, they need to be &lt;strong&gt;stored, versioned, and re-executed&lt;/strong&gt; as systems evolve.&lt;/p&gt;

&lt;p&gt;Prompt changes, model upgrades, workflow modifications, tool additions, and retrieval improvements can all introduce unexpected regressions. Without a persistent evaluation suite, it becomes difficult to determine whether changes are improving the system or silently degrading it.&lt;/p&gt;

&lt;p&gt;A test case becomes a &lt;strong&gt;reusable artifact&lt;/strong&gt; containing:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Inputs&lt;/li&gt;
&lt;li&gt;Context&lt;/li&gt;
&lt;li&gt;Expected behavior&lt;/li&gt;
&lt;li&gt;Evaluation criteria&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Once persisted, these datasets become &lt;strong&gt;regression suites&lt;/strong&gt; that can be executed repeatedly throughout development and deployment.&lt;/p&gt;




&lt;h2&gt;
  
  
  Observability and Self-Learning is the Next Frontier
&lt;/h2&gt;

&lt;p&gt;As I worked through these ideas, one thing became increasingly clear:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Even with comprehensive evaluation datasets, no test suite can realistically capture every scenario an autonomous agent will encounter in the real world.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;p&gt;This is why modern evaluation frameworks place so much emphasis on &lt;strong&gt;tracing and metrics&lt;/strong&gt;. The &lt;strong&gt;same signals used to evaluate agents during testing can also be used to understand how they behave in production&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;What I find particularly interesting is that these capabilities are &lt;strong&gt;not just useful for testing&lt;/strong&gt;. They are &lt;strong&gt;foundational building blocks for systems that can learn from their own behavior&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;That's a topic I'll explore in future articles, but it starts with a simple idea:&lt;/p&gt;

&lt;blockquote&gt;
&lt;p&gt;Before a system can learn from its behavior, we first need to be able to measure it.&lt;/p&gt;
&lt;/blockquote&gt;




&lt;h2&gt;
  
  
  Key Takeaways
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Traditional software testing breaks down with LLM-driven agents due to probabilistic behavior&lt;/li&gt;
&lt;li&gt;Agent evaluation requires measuring across 6+ dimensions (outcome, operational, reliability, quality, governance, behavioral)&lt;/li&gt;
&lt;li&gt;Three primary measurement approaches: mathematical metrics, specialized models, and LLM-as-a-Judge&lt;/li&gt;
&lt;li&gt;Traditional testing remains critical for deterministic system layers&lt;/li&gt;
&lt;li&gt;Modern frameworks converge on tracing, metrics, and persistent test datasets&lt;/li&gt;
&lt;li&gt;The future lies in observability-driven development where testing metrics inform production behavior&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published on &lt;a href="https://susheemk.substack.com/p/from-validating-code-to-evaluating" rel="noopener noreferrer"&gt;Substack&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;




&lt;p&gt;The promise of AI agents is flexibility. They can solve problems we never anticipated. The challenge is control. Traditional testing can't capture every scenario an agent will encounter. That's why modern evaluation frameworks are moving toward observability: continuous measurement of agent behavior across outcome, reliability, cost, and safety dimensions.&lt;/p&gt;

&lt;p&gt;We're building &lt;a href="https://theagentplane.github.io" rel="noopener noreferrer"&gt;Agentplane&lt;/a&gt; to provide the infrastructure layer for this shift. &lt;a href="https://github.com/theagentplane/tokenops" rel="noopener noreferrer"&gt;TokenOps&lt;/a&gt; is our FinOps control plane that enforces token budgets and governance policies across agent delegation trees, giving you real-time cost attribution and risk enforcement. &lt;a href="https://github.com/theagentplane/chronicle" rel="noopener noreferrer"&gt;Chronicle&lt;/a&gt; captures execution traces and behavioral signals from every agent run, turning observability data into actionable insights for improvement and compliance.&lt;/p&gt;

&lt;p&gt;Together, they close the gap between testing and production: measure your agents during development, enforce policies at runtime, and learn from real-world behavior.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>testing</category>
      <category>agents</category>
      <category>software</category>
    </item>
    <item>
      <title>How coding agents like Cursor quietly cut input costs by reusing KV states across turns — and what actually breaks the cache</title>
      <dc:creator>Susheem Koul</dc:creator>
      <pubDate>Thu, 30 Jul 2026 19:11:26 +0000</pubDate>
      <link>https://dev.to/susheem-k/how-coding-agents-like-cursor-quietly-cut-input-costs-by-reusing-kv-states-across-turns-and-what-49fe</link>
      <guid>https://dev.to/susheem-k/how-coding-agents-like-cursor-quietly-cut-input-costs-by-reusing-kv-states-across-turns-and-what-49fe</guid>
      <description>&lt;h2&gt;
  
  
  Why my Cursor bill looked weird
&lt;/h2&gt;

&lt;p&gt;I was poking around my usage dashboard in Cursor and noticed a metric I'd never paid attention to before: &lt;strong&gt;Cache Read&lt;/strong&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Date            Type       Model    Tokens
May 2, 06:50 PM  Included   auto     898K   → Cache Read
May 2, 06:49 PM  Included   auto     1.4M   → Cache Write
May 2, 06:47 PM  Included   auto     346.8K → Input / Output
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Almost 99% of my input tokens for that session were served from cache. That sent me down a rabbit hole on what prompt caching actually &lt;em&gt;is&lt;/em&gt;, and why it matters so much for agentic coding tools specifically. Here's what I found.&lt;/p&gt;

&lt;h2&gt;
  
  
  What prompt caching actually is
&lt;/h2&gt;

&lt;p&gt;Agentic coding tools ship a &lt;em&gt;lot&lt;/em&gt; of context on every single call — system instructions, workspace rules, file contents, conversation history, the works. LLMs are stateless by default, so naively, all of that gets reprocessed from scratch on every request.&lt;/p&gt;

&lt;p&gt;Prompt caching short-circuits this. It saves the intermediate computation from the transformer's prefill stage — specifically the Key/Value states — so that when a later request shares the same leading sequence of tokens, the model reuses those states instead of recomputing them.&lt;/p&gt;

&lt;p&gt;One important caveat: &lt;strong&gt;this only applies to input tokens.&lt;/strong&gt; Output generation is still sequential, token-by-token, and gets no speedup from caching.&lt;/p&gt;

&lt;h2&gt;
  
  
  How the payload is actually built
&lt;/h2&gt;

&lt;p&gt;Coding agents don't just fire your question at the model. They assemble a large composite prompt, and — critically — they structure it with the most static content first and the most volatile content last:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight xml"&gt;&lt;code&gt;&lt;span class="nt"&gt;&amp;lt;system_instructions&amp;gt;&lt;/span&gt; You are an expert coding assistant... &lt;span class="nt"&gt;&amp;lt;/system_instructions&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;project_rules&amp;gt;&lt;/span&gt; Use functional React components... &lt;span class="nt"&gt;&amp;lt;/project_rules&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;file_context_1&amp;gt;&lt;/span&gt; ... &lt;span class="nt"&gt;&amp;lt;/file_context_1&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;file_context_2&amp;gt;&lt;/span&gt; ... &lt;span class="nt"&gt;&amp;lt;/file_context_2&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;conversation_history&amp;gt;&lt;/span&gt; ... &lt;span class="nt"&gt;&amp;lt;/conversation_history&amp;gt;&lt;/span&gt;
&lt;span class="nt"&gt;&amp;lt;user_input&amp;gt;&lt;/span&gt; Refactor this function to handle edge cases. &lt;span class="nt"&gt;&amp;lt;/user_input&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Once that sequence hits the model, the transformer tokenizes it and starts computing attention. For every token, it derives a Key (K) and a Value (V) — essentially a map of how that token relates to every other token before it. This is the expensive part: attention computation scales quadratically with sequence length, so a big context window (lots of files, long history) gets disproportionately costly to prefill.&lt;/p&gt;

&lt;p&gt;The model doesn't refer back to your raw text when generating a response — it refers to these computed KV matrices. That's the artifact caching actually preserves.&lt;/p&gt;

&lt;p&gt;Providers typically charge less for input tokens than output tokens, but agentic coding tools push such enormous input payloads that shaving cost off the input side is still very much worth doing.&lt;/p&gt;

&lt;h2&gt;
  
  
  Turn 1 vs. every turn after
&lt;/h2&gt;

&lt;p&gt;Here's the flow, roughly:&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Turn 1 (cold start):&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Build the prompt: system instructions + project rules + file context + your first message.&lt;/li&gt;
&lt;li&gt;No cache exists yet → full prefill. The model computes KV states for &lt;em&gt;every&lt;/em&gt; token (the expensive O(n²) pass).&lt;/li&gt;
&lt;li&gt;Those KV states get written to cache storage.&lt;/li&gt;
&lt;li&gt;Generation proceeds normally.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;&lt;strong&gt;Turn 2...N (warm):&lt;/strong&gt;&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Build the new prompt: same system instructions + same files + prior conversation + your new message.&lt;/li&gt;
&lt;li&gt;Check for a prefix cache hit against system + context.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Hit:&lt;/strong&gt; read the cached KV states for the unchanged prefix, and only compute &lt;em&gt;new&lt;/em&gt; KV states for the incremental conversation turn + your latest input.&lt;/li&gt;
&lt;li&gt;Concatenate cached + new KV states, then generate.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;So instead of re-running the full O(n²) prefill on every message, only the marginal new tokens get freshly processed. That's the whole trick, and it's why long coding sessions don't get proportionally slower and more expensive turn after turn.&lt;/p&gt;

&lt;p&gt;Most major providers now do this optimization implicitly under the hood. Google's Vertex AI docs on &lt;a href="https://docs.cloud.google.com/vertex-ai/generativeai/docs/context-cache/context-cache-overview" rel="noopener noreferrer"&gt;context caching&lt;/a&gt; are a good primary-source reference if you want to see how one provider exposes this explicitly.&lt;/p&gt;

&lt;h2&gt;
  
  
  What breaks the cache
&lt;/h2&gt;

&lt;p&gt;Caching depends entirely on &lt;strong&gt;prefix stability&lt;/strong&gt;. The moment a token diverges from what's cached, everything downstream of that point has to be recomputed. In practice, here's what kills your hit rate:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Dynamic injections&lt;/strong&gt; — a timestamp or other changing value stuffed near the top of the prompt invalidates the prefix on every single request.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Context reordering&lt;/strong&gt; — shuffling the order in which files or context blocks get loaded prevents prefix matching entirely.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Upstream file edits&lt;/strong&gt; — editing a file that sits early in the prompt structure invalidates the cache for that file &lt;em&gt;and everything that comes after it&lt;/em&gt; in the sequence.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Most systems rely on exact token-prefix matches, though some providers (Google's docs linked above, for instance) expose more explicit cache handles or partial-reuse mechanisms.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part that confused me: why appending doesn't break the cache
&lt;/h2&gt;

&lt;p&gt;This is the part that seemed contradictory at first. If attention connects every token to every other token, shouldn't adding a new message at the &lt;em&gt;end&lt;/em&gt; of the prompt ripple backward and change how the &lt;em&gt;earlier&lt;/em&gt; tokens (system instructions, file context) get attended to?&lt;/p&gt;

&lt;p&gt;It would — if the model attended in both directions. But modern generative LLMs are &lt;strong&gt;decoder-only&lt;/strong&gt; and use &lt;strong&gt;causal (masked) self-attention&lt;/strong&gt;: a token can only attend to tokens that came &lt;em&gt;before&lt;/em&gt; it, never after.&lt;/p&gt;

&lt;p&gt;That one architectural detail is what makes caching possible at all. We're not caching a fragile, bidirectional global attention matrix — we're caching the &lt;strong&gt;KV tensors of the prefix&lt;/strong&gt;. Those K and V states for the early context are computed once and never change, no matter what you append afterward. New tokens just generate their own Queries and attend against the already-cached Keys of everything before them.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where this goes next
&lt;/h2&gt;

&lt;p&gt;Prompt caching solves the "don't recompute what hasn't changed" problem. It doesn't solve the separate problem of &lt;em&gt;what to put in the context window in the first place&lt;/em&gt;. Instead of stuffing every file into the prompt and hoping for a cache hit, coding agents increasingly rely on retrieval — RAG-style embeddings over the codebase — to selectively pull in only the relevant slices of code.&lt;/p&gt;

&lt;p&gt;That's next on my list to dig into: how coding tools build queryable views of a codebase with embeddings, and what that does to the cost equation.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;If you want the deeper transformer-architecture background behind KV caching and attention, &lt;a href="https://www.3blue1brown.com" rel="noopener noreferrer"&gt;3Blue1Brown's "Large Language Models explained briefly"&lt;/a&gt; is a genuinely excellent, low-jargon primer.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Originally posted on &lt;a href="https://susheemk.substack.com/p/prompt-caching-and-context-optimization" rel="noopener noreferrer"&gt;Susheem's Jottings&lt;/a&gt; — cross-posting here for the dev.to crowd.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>webdev</category>
      <category>llm</category>
    </item>
  </channel>
</rss>
