<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Mayur Agnani</title>
    <description>The latest articles on DEV Community by Mayur Agnani (@theaicommit).</description>
    <link>https://dev.to/theaicommit</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F2280575%2F3c36a1b6-308f-429b-a971-c39346a9c238.jpg</url>
      <title>DEV Community: Mayur Agnani</title>
      <link>https://dev.to/theaicommit</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/theaicommit"/>
    <language>en</language>
    <item>
      <title>How Sorting Your Prompts Speeds Up a Mixture-of-Experts Batch Job</title>
      <dc:creator>Mayur Agnani</dc:creator>
      <pubDate>Wed, 21 Oct 2026 10:50:14 +0000</pubDate>
      <link>https://dev.to/theaicommit/how-sorting-your-prompts-speeds-up-a-mixture-of-experts-batch-job-387o</link>
      <guid>https://dev.to/theaicommit/how-sorting-your-prompts-speeds-up-a-mixture-of-experts-batch-job-387o</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Engineer's view.&lt;/strong&gt; This is a batch job that touches records in arrival order. Each record needs a different index page, so the cache keeps reloading pages. Sort by key and each page loads once per batch. A mixture-of-experts model loads expert weights the same way, so a mixed batch pays for reads nobody shares.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TL;DR.&lt;/strong&gt; A mixture-of-experts model reads only the experts the batch asked for, and similar prompts ask for the same ones. Sort an offline job by topic, and each step reads less, so the job finishes sooner.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Explain Like I'm 5
&lt;/h2&gt;

&lt;p&gt;A library has a hundred shelves. A helper runs to fetch books for everyone at one table. If the table holds a cook, a pilot and a poet, the helper runs to many shelves each round. If everyone at the table is a cook, they mostly want the same cooking shelf. The helper makes fewer trips, and the table gets its books sooner. Nobody reads faster. The seating just changed.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Problem
&lt;/h2&gt;

&lt;p&gt;You have shipped this bug before, in a system with no AI in it.&lt;/p&gt;

&lt;p&gt;A nightly job processes a million records in the order they arrived. Each record looks up a row in a big table. Records from the same customer sit on the same index pages, but arrival order scatters them. So the database keeps loading a page, evicting it, and loading it again an hour later. The job is slow, and the code is fine. Someone sorts the input by customer ID, and the job runs twice as fast. Nothing about the work changed. Only the order did.&lt;/p&gt;

&lt;p&gt;A mixture-of-experts model has the same bug inside every decode step. Each layer holds many small blocks of weights, called &lt;strong&gt;experts&lt;/strong&gt;. Each token uses only a few of them. Qwen3-30B-A3B has 128 experts per layer, and each token picks 8. But a server runs many requests together as one batch. On each step it must read every expert that any request in the batch picked.&lt;/p&gt;

&lt;p&gt;That read is the expensive part. Producing one token per request is limited by how fast the GPU reads weights from memory, not by arithmetic. A random batch of 32 Qwen requests touches about 94 of the 128 experts in a layer. Code prompts favor one set of experts, French prompts another. Mix them, and the step reads both sets.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The fix: before an offline batch job, sort the prompts so similar ones run together. Then each step reads fewer experts.&lt;/strong&gt; Hugging Face measured this on unchanged vLLM and got 5 to 17% more throughput.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Fix: Group Similar Prompts Before the Batch Job
&lt;/h2&gt;

&lt;p&gt;New to how a model produces text one token at a time? Start at AI basics → &lt;a href="https://theaicommit.com/#learn/tokens-and-sampling" rel="noopener noreferrer"&gt;Tokens and sampling&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does the order of a list change what the GPU reads?
&lt;/h3&gt;

&lt;p&gt;Because the server fills its batch from the front of the list. vLLM admits waiting requests first come, first served. So neighbors in your list end up decoding side by side. Sort the list, and each batch is mostly one topic.&lt;/p&gt;

&lt;p&gt;Topics really do prefer their own experts. In one layer of Qwen3-30B-A3B, a topic's 32 favorite experts took 68 to 80% of its routing decisions. Even routing would give them 25%.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F588637sft2k4stdagejt.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F588637sft2k4stdagejt.png" alt="Same prompts, two orders, different reads per step" width="800" height="306"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h3&gt;
  
  
  How much does one expert cost?
&lt;/h3&gt;

&lt;p&gt;About 2.9 microseconds per layer on an AMD MI300X. That is the time to read about 9.4 MB of expert weights. The MoE layer's time rose in a straight line with the number of distinct experts the batch touched.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why was the measured gain smaller than the reads saved?
&lt;/h3&gt;

&lt;p&gt;Sorted Qwen batches read 19 to 25% fewer experts, which predicts a 20 to 27% speedup. The measured gain was about half that. Only the expert layers got cheaper. Attention, sampling and scheduling cost the same as before.&lt;/p&gt;

&lt;h3&gt;
  
  
  Do I need topic labels?
&lt;/h3&gt;

&lt;p&gt;No. Embed each prompt with a small embedding model, cluster the vectors with k-means, and sort by cluster. On unlabeled chat data, this still gave Qwen about 5% more throughput.&lt;/p&gt;

&lt;h2&gt;
  
  
  What This Means for You
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;When this matters.&lt;/strong&gt; You run offline jobs on a mixture-of-experts model that you host yourself. Think evals, synthetic data, labeling, summarizing a backlog. Qwen3 MoE, gpt-oss, DeepSeek and Mixtral are all this kind of model. The job generates many tokens per prompt, and nobody waits on any single answer.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How it affects you.&lt;/strong&gt; The win is GPU hours. A ten-hour job on a rented node can finish in about nine. Your outputs will also shift a little. In the Hugging Face run, 92% of greedy outputs matched between sorted and shuffled order. Two shuffled runs matched each other 90% of the time. That drift is ordinary batch nondeterminism, the effect from &lt;a href="https://theaicommit.com/#2026-09-02" rel="noopener noreferrer"&gt;How the Same Model Gives Two Different Answers&lt;/a&gt;. Sorting does not add new drift.&lt;/p&gt;

&lt;p&gt;The gain also depends on where your batch size sits. The simulation in &lt;a href="https://github.com/magna56/aI_daily_run/blob/main/2026-10-21/code_example.py" rel="noopener noreferrer"&gt;&lt;code&gt;code_example.py&lt;/code&gt;&lt;/a&gt; shows the shape: zero at batch 1, largest at small and middle batches, fading once a one-topic batch reads nearly every expert.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvgx9dqcvceu56y1lfc55.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fvgx9dqcvceu56y1lfc55.png" alt="Throughput gain from sorting, Qwen3-30B-A3B, 6 topics" width="800" height="310"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What to do about it.&lt;/strong&gt; Today, look at your next offline job's input file. If it already has a dataset, task or language column, sort by it before you submit. That costs one line. If it has no labels, add the clustering step below. Then run the A/B test in Implementing It before you trust any percentage, including the one in this article.&lt;/p&gt;

&lt;h2&gt;
  
  
  Implementing It
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The change.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Three roles touch this. Whoever builds the input file groups the prompts. Whoever runs the job submits them in that order and restores the original order. Whoever owns the cluster checks the gain is real. The serving side changes nothing for an offline job.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Role 1, grouping.&lt;/em&gt; Use labels when you have them. Otherwise, cluster embeddings. The source used Qwen3-Embedding-0.6B and 16 clusters:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;numpy&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sentence_transformers&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;SentenceTransformer&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;sklearn.cluster&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;KMeans&lt;/span&gt;

&lt;span class="n"&gt;emb&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;SentenceTransformer&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Qwen/Qwen3-Embedding-0.6B&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;encode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;prompts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;normalize_embeddings&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;batch_size&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;labels&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;KMeans&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;n_clusters&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;16&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n_init&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;auto&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;random_state&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;fit_predict&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;emb&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;order&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;np&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;argsort&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;kind&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;stable&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;  &lt;span class="c1"&gt;# stable keeps file order inside a cluster
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Keep the embedding model small. It runs once over the file, before the expensive model loads.&lt;/p&gt;

&lt;p&gt;You may not be able to sort the whole file at once, for example when you stream it in pieces. Then sort inside each piece. At batch 32, sorting within chunks of 128 prompts gave +11.7%, and chunks of 256 gave +14.8%, against +17.3% for a full sort. Make each piece several batches long, so every batch can fill from one cluster.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Role 2, the batch job.&lt;/em&gt; vLLM returns outputs in the order you submitted. So submit in sorted order, then put each output back where its prompt came from:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="c1"&gt;# before
&lt;/span&gt;&lt;span class="n"&gt;outs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;llm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# after
&lt;/span&gt;&lt;span class="n"&gt;outs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;llm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generate&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;prompts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;results&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;prompts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;pos&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;enumerate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;order&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;results&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;outs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;pos&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Skip the restore step and every downstream join is wrong. You will see it as answers attached to the wrong question, not as an error.&lt;/p&gt;

&lt;p&gt;Your batch size is &lt;code&gt;max_num_seqs&lt;/code&gt; on the &lt;code&gt;LLM&lt;/code&gt; constructor. It decides where you sit on the curve above, so record it with every result.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Role 3, measuring.&lt;/em&gt; Run the same prompts in both orders and compare tokens per second. Force equal output lengths so only the order differs:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;
&lt;span class="n"&gt;params&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;SamplingParams&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;temperature&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_tokens&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ignore_eos&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="bp"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;tok_per_s&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;outs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;llm&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;generate&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;batch&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;params&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;outputs&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;token_ids&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;o&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;outs&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;perf_counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Alternate the runs: shuffled, sorted, shuffled, sorted. A single pair can be skewed by a warm cache. The author's full harness is &lt;code&gt;vllm_bench.py&lt;/code&gt; in the sort-your-prompts repository on GitHub. It had no license file when this was written, so read it for the method.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How you know it worked.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;The sorted runs should beat the shuffled runs on every repeat, not just on average. On a mixture-of-experts model with labeled topics, expect something near 10%. Then run the same test on a dense model of similar size as a control. It should show no gain, because a dense model reads every weight on every step. If the dense model "improves" too, your measurement is noise. Last, compare outputs. The share that differs between sorted and shuffled should match the share that differs between two shuffled runs. If it is much higher, check the restore step.&lt;/p&gt;

&lt;h2&gt;
  
  
  When Sorting Prompts Is the Wrong Tool
&lt;/h2&gt;

&lt;p&gt;When someone is waiting. Grouping means holding a request until similar ones arrive. In online serving, that queueing delay costs more than the step time it saves.&lt;/p&gt;

&lt;p&gt;When the model is dense. Every weight is read on every step, whatever the order. A dense model like Llama 3 gains nothing.&lt;/p&gt;

&lt;p&gt;When the work is mostly prefill. A long prompt with a short answer spends its time reading the prompt. Thousands of prompt tokens touch nearly every expert anyway. A classifier that emits one label per document is this case.&lt;/p&gt;

&lt;p&gt;When batches are very large. Once even a one-topic batch touches almost every expert, there is nothing left to save. On unlabeled chat data, gpt-oss-120b at batch 128 showed no measurable gain.&lt;/p&gt;

&lt;p&gt;When the model has few experts. A coarse model with 32 experts per layer saved 8 to 11% of reads, against 19 to 25% for 128 experts.&lt;/p&gt;

&lt;p&gt;Before you adopt it, ask:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Is my model a mixture of experts, and is this job decode-heavy?&lt;/li&gt;
&lt;li&gt;What batch size does my job really run at?&lt;/li&gt;
&lt;li&gt;Did a dense control show zero gain on the same A/B test?&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Glossary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;experts&lt;/strong&gt; — small blocks of weights in a layer; each token is routed to a few of them.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;mixture-of-experts&lt;/strong&gt; — a model whose layers hold many experts and use only a few per token.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;decode step&lt;/strong&gt; — one round in which every request in the batch produces its next token.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;batch&lt;/strong&gt; — the set of requests the server runs together on each step.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;k-means&lt;/strong&gt; — a clustering method that groups vectors around a chosen number of centers.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;batch nondeterminism&lt;/strong&gt; — small output changes caused only by which requests shared a batch.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://huggingface.co/blog/badaoui/sort-your-prompts" rel="noopener noreferrer"&gt;Sort your prompts: a free speed-up for MoE batch inference&lt;/a&gt; — Hugging Face blog (Abdennacer Badaoui)&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/Abdennacer-Badaoui/sort-your-prompts" rel="noopener noreferrer"&gt;Abdennacer-Badaoui/sort-your-prompts&lt;/a&gt; — GitHub, the author's code&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://huggingface.co/Qwen/Qwen3-30B-A3B" rel="noopener noreferrer"&gt;Qwen3-30B-A3B model card&lt;/a&gt; — Qwen team, Hugging Face&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://huggingface.co/blog/moe" rel="noopener noreferrer"&gt;Mixture of Experts Explained&lt;/a&gt; — Hugging Face blog&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://theaicommit.com/2026-10-21-how-sorting-your-prompts-speeds-up-a-mixture-of-experts/" rel="noopener noreferrer"&gt;theaicommit.com&lt;/a&gt;, a daily AI engineering lab for software engineers. The original has an &lt;a href="https://theaicommit.com/2026-10-21-how-sorting-your-prompts-speeds-up-a-mixture-of-experts/" rel="noopener noreferrer"&gt;interactive version&lt;/a&gt; you can play with, and the full runnable code is on &lt;a href="https://github.com/magna56/aI_daily_run/blob/main/2026-10-21/code_example.py" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>performance</category>
      <category>machinelearning</category>
    </item>
    <item>
      <title>How to Stop a Coding Agent and an AI Reviewer From Looping on One Pull Request</title>
      <dc:creator>Mayur Agnani</dc:creator>
      <pubDate>Tue, 20 Oct 2026 11:29:01 +0000</pubDate>
      <link>https://dev.to/theaicommit/how-to-stop-a-coding-agent-and-an-ai-reviewer-from-looping-on-one-pull-request-2p1k</link>
      <guid>https://dev.to/theaicommit/how-to-stop-a-coding-agent-and-an-ai-reviewer-from-looping-on-one-pull-request-2p1k</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Engineer's view.&lt;/strong&gt; This is "retry until green" on a flaky test suite. The job stops at the first green run, so noise decides when it ends, and one green run proves little. An agent and an AI reviewer stop the same way: at the first review with nothing to fix. It costs rounds and missed bugs.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TL;DR.&lt;/strong&gt; An agent and an AI reviewer stop looping only when a review finds nothing to fix. Let only new, important problems on changed lines block, and stop after two clean reviews in a row.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Explain Like I'm 5
&lt;/h2&gt;

&lt;p&gt;You ask a friend to check your story for mistakes. Every time they find something, you fix it and hand it back. Some of their notes are real mistakes. Some are just "I'd use a different word." Each fix gives them a fresh page to find new little things on. So you go back and forth all afternoon. Then one time they glance and say "looks fine," and you stop. But a glance can miss a real mistake. Better rules: only fix real mistakes, and stop when they've read it carefully twice and found nothing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Problem
&lt;/h2&gt;

&lt;p&gt;You have shipped this loop before, in CI.&lt;/p&gt;

&lt;p&gt;A test suite has a few flaky tests. Someone adds a retry: run the suite until it passes, up to five times. Two things go wrong. Builds take longer, because the flaky tests keep failing for no reason. And a real failure that only shows up half the time now gets through. One lucky green run is enough. The stop rule was "the first green run," and noise decided when that came.&lt;/p&gt;

&lt;p&gt;A coding agent working with an AI reviewer runs the same loop. The reviewer reads the pull request and leaves comments. The agent fixes every one, because to an agent a comment is an instruction. Each fix is a new commit, so the reviewer runs again. The loop ends at the first review that finds nothing to fix.&lt;/p&gt;

&lt;p&gt;Anthropic's Claude Code team, where Claude writes about 90% of the code, measured this. Before they changed their rules, 8.9% of internal pull requests needed five or more review rounds. One pull request ran more than fifty rounds overnight. About 59% of what the reviewer kept flagging in later rounds was code that was already there before the change.&lt;/p&gt;

&lt;p&gt;The long tail costs CI time and attention. The quieter cost is the one the flaky suite taught you: a single clean review is weak evidence when the reviewer misses half the bugs on any one pass.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Decide what is allowed to block, and decide when the loop may stop. Block only new, important findings on lines the change touched. Stop after two clean reviews in a row. After three fix rounds, hand the pull request to a person.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The Fix: Let Only Real Findings Block, and Require Two Clean Passes
&lt;/h2&gt;

&lt;p&gt;New to how an agent works through a task? Start at AI basics → &lt;a href="https://theaicommit.com/#learn/the-agent-loop" rel="noopener noreferrer"&gt;The agent loop&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why does the loop run so long?
&lt;/h3&gt;

&lt;p&gt;Because the stop rule counts every comment. A &lt;strong&gt;review pass&lt;/strong&gt; is one full read of the pull request by the reviewer. A &lt;strong&gt;fix round&lt;/strong&gt; is one commit the agent makes in answer to it. The loop stops at the first pass with nothing to fix.&lt;/p&gt;

&lt;p&gt;Three kinds of comment keep that from happening. Nits are new on every pass, because the reviewer samples them fresh. Findings in old code reappear, because the old code is still there. And the same finding can come back in new words. The agent fixes all of them, and each fix starts another round.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why not just cap the rounds, or re-review only the new lines?
&lt;/h3&gt;

&lt;p&gt;Both cut rounds, and both lose bugs. Each full pass is another independent look at the real bugs the last pass missed. In the simulation in &lt;a href="https://github.com/magna56/aI_daily_run/blob/main/2026-10-20/code_example.py" rel="noopener noreferrer"&gt;&lt;code&gt;code_example.py&lt;/code&gt;&lt;/a&gt;, the reviewer's recall is 50%: it catches each real bug half the time per pass. Cutting the noise but stopping at one clean pass let 24.4% of bugs escape. The noisy loop let 13.1% escape. Its extra rounds were accidental second looks.&lt;/p&gt;

&lt;h3&gt;
  
  
  So what should block, and when should it stop?
&lt;/h3&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2t890alqnvq56nfrf57s.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2F2t890alqnvq56nfrf57s.png" alt="One review pass, three outcomes" width="800" height="386"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Only a &lt;strong&gt;blocking finding&lt;/strong&gt; starts a fix round. That means new, tagged important, and on a line the pull request changed. A &lt;strong&gt;pre-existing finding&lt;/strong&gt; goes on a list once. A finding is identified by its file, line, and rule, so a repeat is dropped. A &lt;strong&gt;clean pass&lt;/strong&gt; must happen twice in a row before the gate passes. The second clean pass costs a review, but no agent commit and no CI run.&lt;/p&gt;

&lt;h2&gt;
  
  
  What This Means for You
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;When this matters.&lt;/strong&gt; You run a coding agent that answers review comments on its own. That might be the Claude Code GitHub Action, Copilot's coding agent with Copilot review, or your own loop. It matters most on large pull requests. The Claude Code team found most pull requests over 1,000 lines get findings, while most small ones come back clean.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How it affects you.&lt;/strong&gt; The first thing you notice is CI cost and slow merges. The thing you don't notice is escapes. Loosening the loop to save rounds can quietly double the bugs that get through. The agent's fix commits are the expensive part. Each one reruns CI, and each one can add a bug of its own. In the simulation, the two-clean-pass gate cut agent fix rounds from 2.07 to 1.01 per pull request. No pull request needed five rounds. Escapes stayed at 13.1%, the same as the noisy loop. The cost: 13.4% of pull requests went to a person after three rounds.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flyiddjflkdlcvidmglxg.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Flyiddjflkdlcvidmglxg.png" alt="Bugs that escaped review, same 4,000 simulated pull requests" width="799" height="272"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What to do about it.&lt;/strong&gt; Today, count how many review rounds your last twenty agent pull requests took. Look for rounds where the agent fixed only nits or old code; those are the rounds the gate removes. Then write down what is allowed to block. Change the stop rule last, and only together with the blocking rule.&lt;/p&gt;

&lt;h2&gt;
  
  
  Implementing It
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The change.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Three roles touch this. Whoever configures the reviewer decides what it tags important. Whoever owns CI runs the gate. Whoever writes the agent's instructions stops it from fixing what does not block.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Role 1, the reviewer's configuration.&lt;/em&gt; If you use Claude's managed code review, put this in &lt;code&gt;REVIEW.md&lt;/code&gt; at the repo root. Other reviewers take the same text as part of their prompt:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;&lt;span class="gu"&gt;## What counts as Important here&lt;/span&gt;
&lt;span class="p"&gt;-&lt;/span&gt; A bug on a line this pull request changed, with file:line evidence.
&lt;span class="p"&gt;-&lt;/span&gt; Not style, naming or cleanup. Tag those Nit.
&lt;span class="p"&gt;-&lt;/span&gt; Code the pull request did not touch: tag Pre-existing, never Important.
Start your summary with the count of Important findings.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ask the reviewer to emit its findings as JSON: path, line, rule, severity. The gate reads that. Keep the rule names stable from pass to pass, because the gate matches repeats on them.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Role 2, CI.&lt;/em&gt; The gate is a required status check on every push. It keeps a small state file per pull request, so it can count clean passes and fix rounds across pushes:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;decide&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;findings&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;blocking&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;findings&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
                &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;severity&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;important&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;changed&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;blocking&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;clean_streak&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pass&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;clean_streak&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;REQUIRED_CLEAN&lt;/span&gt; &lt;span class="k"&gt;else&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;review&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;clean_streak&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fix_rounds&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;MAX_FIX_ROUNDS&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;escalate&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
    &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;fix_rounds&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;fix&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;self.new()&lt;/code&gt; drops any finding whose path, line, and rule were already seen. On "review", run the reviewer again without waiting for a commit. On "escalate", fail the check and request a human reviewer. Get the changed lines from &lt;code&gt;git diff --unified=0&lt;/code&gt; against the merge base, so a rebase does not count old lines as new. Store the state as a hidden comment on the pull request or in the CI cache. Either one survives between pushes.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Role 3, the agent's instructions.&lt;/em&gt; Add this to &lt;code&gt;CLAUDE.md&lt;/code&gt;, or to whatever file your agent reads:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight markdown"&gt;&lt;code&gt;When you answer review comments, fix only findings the review gate lists as blocking.
Do not fix Nit or Pre-existing findings in this pull request. Do not edit lines
outside the diff to satisfy a comment.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Without this, the agent fixes nits anyway, and every nit still costs a commit.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How you know it worked.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Before you switch, record three numbers over twenty pull requests: fix rounds per pull request, the share needing five or more rounds, and reverts within a week of merge. After two weeks, rounds should fall and the five-round tail should vanish. Escalations should stay near one in ten; far above that, the important tag is too loose. Reverts must not rise. If they do, you have only moved the work, so raise the required clean passes before anything else.&lt;/p&gt;

&lt;h2&gt;
  
  
  When a Two-Clean-Pass Gate Is the Wrong Tool
&lt;/h2&gt;

&lt;p&gt;When two passes are not independent. The second pass helps only if it can miss different things than the first. A reviewer at temperature zero, with the same prompt and the same context, may return the same answer twice. Then the second clean pass proves nothing. Vary something between passes: a second model, a different focus, or parallel reviewers that vote.&lt;/p&gt;

&lt;p&gt;When a person writes the code. A human author pushes back on nits and rarely fixes all of them. The loop that this gate controls mostly doesn't form.&lt;/p&gt;

&lt;p&gt;When the change is small and low risk. A docs fix or a flagged, additive change can pass on one clean review. Match the passes to the risk, as the Claude Code team does with its review tiers.&lt;/p&gt;

&lt;p&gt;When nobody reads the escalations. A gate that sends 13% of pull requests to a person needs a person. Otherwise those pull requests just sit, and engineers learn to route around the check.&lt;/p&gt;

&lt;p&gt;Before you adopt it, ask:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Do my reviewer's passes actually differ from each other?&lt;/li&gt;
&lt;li&gt;Who picks up a pull request after three fix rounds?&lt;/li&gt;
&lt;li&gt;Do I know my revert rate today?&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Glossary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;review pass&lt;/strong&gt; — one full read of the pull request by the AI reviewer.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;fix round&lt;/strong&gt; — one commit the agent makes in answer to a review pass.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;blocking finding&lt;/strong&gt; — a new finding, tagged important, on a line the pull request changed.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;pre-existing finding&lt;/strong&gt; — a finding in code the pull request did not touch; listed, never blocking.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;clean pass&lt;/strong&gt; — a review pass with no blocking findings.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;recall&lt;/strong&gt; — the share of real bugs one review pass catches.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://claude.dev/blog/solving-code-review-at-anthropic/" rel="noopener noreferrer"&gt;How the Claude Code team does code review&lt;/a&gt; — Anthropic, claude.dev (Addy Osmani, Morgan Westlee Lunt, Qing Wang)&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://code.claude.com/docs/en/code-review" rel="noopener noreferrer"&gt;Code Review — Claude Code Docs&lt;/a&gt; — Anthropic, Claude Code documentation&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/anthropics/claude-code-action" rel="noopener noreferrer"&gt;anthropics/claude-code-action&lt;/a&gt; — Anthropic, MIT license&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://www.anthropic.com/engineering/redesigning-source-control" rel="noopener noreferrer"&gt;Redesigning source control at Anthropic&lt;/a&gt; — Anthropic Engineering&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://theaicommit.com/2026-10-20-how-to-stop-a-coding-agent-and-an-ai-reviewer-from-looping/" rel="noopener noreferrer"&gt;theaicommit.com&lt;/a&gt;, a daily AI engineering lab for software engineers. The original has an &lt;a href="https://theaicommit.com/2026-10-20-how-to-stop-a-coding-agent-and-an-ai-reviewer-from-looping/" rel="noopener noreferrer"&gt;interactive version&lt;/a&gt; you can play with, and the full runnable code is on &lt;a href="https://github.com/magna56/aI_daily_run/blob/main/2026-10-20/code_example.py" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>productivity</category>
      <category>testing</category>
      <category>devops</category>
    </item>
    <item>
      <title>How to Make Your Agent Check the Records It Changed Before It Says Done</title>
      <dc:creator>Mayur Agnani</dc:creator>
      <pubDate>Tue, 20 Oct 2026 01:14:16 +0000</pubDate>
      <link>https://dev.to/theaicommit/how-to-make-your-agent-check-the-records-it-changed-before-it-says-done-36p4</link>
      <guid>https://dev.to/theaicommit/how-to-make-your-agent-check-the-records-it-changed-before-it-says-done-36p4</guid>
      <description>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Engineer's view.&lt;/strong&gt; This is an endpoint that returns 200 after writing the wrong value. The handler logged success, and only a look at the row shows the bug. An agent's "done" is that 200: it comes from what its tools returned. Checking the records it changed is the test it never ran.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;TL;DR.&lt;/strong&gt; An agent decides it is done from what its tools returned, and a wrong write returns success just like a right one. Check the records it actually changed against what the task required, before it closes the task.&lt;/p&gt;
&lt;/blockquote&gt;

&lt;h2&gt;
  
  
  Explain Like I'm 5
&lt;/h2&gt;

&lt;p&gt;You ask a friend to put the milk in the fridge. They say "done!" You believe them, because they did walk to the kitchen. Later you find the milk in the cupboard. Your friend was not lying. They opened a door, put the milk behind it, and the door closed fine. Nothing went wrong that they could hear. The only way to know is to open the fridge. So before you say thanks, you look inside.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Problem
&lt;/h2&gt;

&lt;p&gt;You have shipped this bug before, in an API.&lt;/p&gt;

&lt;p&gt;A support tool calls &lt;code&gt;PATCH /orders/A17&lt;/code&gt; to put an order on hold. A mapping bug sends &lt;code&gt;resolved&lt;/code&gt; instead of &lt;code&gt;on_hold&lt;/code&gt;. The endpoint validates the value, writes it, and returns 200. The client logs success. Your dashboard of error rates stays flat. A week later the customer calls, because nobody ever followed up. The bug was never in the transport. It was in the row, and nothing looked at the row.&lt;/p&gt;

&lt;p&gt;An agent that writes to a backend has the same blind spot, at a larger scale. It decides it is done by reading its own transcript. The transcript holds what each tool returned. A write with the wrong value returns "ok" just like a right one.&lt;/p&gt;

&lt;p&gt;Microsoft's ThinkingBox team measured how often this happens. Their example is a customer whose $745 appliance sat in a carrier exception for fifteen days. The agent made nine well-formed tool calls and read the refund policy correctly. Then it closed the ticket as resolved. The policy said it had to stay on hold while the exception was open.&lt;/p&gt;

&lt;p&gt;That was not a rare case. Across 12 models, they ran 121,680 trials and saw 79,853 failures. Of those failures, 67% ended cleanly. The agent changed state, reported no tool error, and said it was done. A check on the transcript would have passed two thirds of the failures.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Check the records the agent changed, not the messages it wrote. Log every write the agent makes, compare those writes to what the task requires, and refuse to close the task until they match.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The Fix: Check the Agent's Writes Before It Closes the Task
&lt;/h2&gt;

&lt;p&gt;New to how an agent decides it is finished? Start at AI basics → &lt;a href="https://theaicommit.com/#learn/the-agent-loop" rel="noopener noreferrer"&gt;The agent loop&lt;/a&gt;.&lt;/p&gt;

&lt;h3&gt;
  
  
  Why doesn't the agent notice on its own?
&lt;/h3&gt;

&lt;p&gt;Because nothing it can see is wrong. The model reads tool results, not your database. If &lt;code&gt;update_ticket&lt;/code&gt; returns &lt;code&gt;{"ok": true}&lt;/code&gt;, the model has no signal that the value it sent was the wrong one. Asking it to "double-check" only makes it reread the same transcript.&lt;/p&gt;

&lt;p&gt;So the check has to read state. Every write the agent makes is an effect: a table, a row, a field, and its value before and after.&lt;/p&gt;

&lt;h3&gt;
  
  
  What does the check compare?
&lt;/h3&gt;

&lt;p&gt;A contract per task type. It lists the fields that must end at a given value and the fields the task may touch. Comparing the effects to the contract finds three kinds of failure:&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqxox6yl71ik8kfk9e0cc.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqxox6yl71ik8kfk9e0cc.png" alt="Same clean transcript, three ways the records can be wrong" width="800" height="370"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;In ThinkingBox's failures, 78% had a wrong field value, 43% had an extra effect, and 25% missed a required one. One run can have more than one.&lt;/p&gt;

&lt;h3&gt;
  
  
  Where does the check go?
&lt;/h3&gt;

&lt;p&gt;On the tool that ends the task, such as &lt;code&gt;close_ticket&lt;/code&gt;. If the check finds a problem, the tool refuses and returns the list of problems as its result. The agent then reads a real signal: which field is wrong, and what it should be. It can fix the field, or hand the ticket to a person.&lt;/p&gt;

&lt;h2&gt;
  
  
  What This Means for You
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;When this matters.&lt;/strong&gt; Any agent that writes to a system of record: tickets, orders, refunds, a CRM, a calendar. It matters most where a wrong write is quiet. A ticket marked resolved sends no alert. A refund issued twice shows up only at month end.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How it affects you.&lt;/strong&gt; Most of your agent's failures will look like successes in your logs. ThinkingBox's authors read the failures as mostly a problem of tool use and recovery, not reasoning. That is good news, because tool use is the part you control.&lt;/p&gt;

&lt;p&gt;There is a second cost. A single run tells you little about the next one. One open-weight model in the study solved 94% of tasks at least once, but only 13% on all 20 tries. Claude Opus 5 solved fewer at least once, yet passed 48% on every try. The &lt;a href="https://github.com/magna56/aI_daily_run/blob/main/2026-10-19/code_example.py" rel="noopener noreferrer"&gt;&lt;code&gt;code_example.py&lt;/code&gt;&lt;/a&gt; file shows why. Two simulated agents both pass about 72% of single runs. One fails a little on every task, and passes all 20 runs on none of them. The other is reliable on most tasks and hopeless on a few, and passes all 20 on 38%.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe200twqgrsdqx4gwhhjn.png" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fe200twqgrsdqx4gwhhjn.png" alt="Tasks passed on all 20 runs, two agents with the same 72% single-run score" width="800" height="236"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;What to do about it.&lt;/strong&gt; Log every write at the tool layer. Write a contract for each task type that changes records. Put the check on the closing tool. Then score your agent by how many tasks pass every run, not by its average.&lt;/p&gt;

&lt;h2&gt;
  
  
  Implementing It
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;The change.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Three roles touch this. Whoever writes the tools records each write. Whoever builds the agent puts the check on the closing tool. Whoever runs the evals scores every task over several clean runs.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Role 1, whoever writes the tools.&lt;/em&gt; Route every state-changing tool through one write function. It records the effect before it returns:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;write&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;row&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="nf"&gt;setdefault&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{})&lt;/span&gt;
    &lt;span class="n"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;effects&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nc"&gt;Effect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;field&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;field&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="n"&gt;row&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;field&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;value&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;ok&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;            &lt;span class="c1"&gt;# all the model will ever see
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Read-only tools skip it. A tool that calls another service records what it asked for, plus the id that service returned. Keep one log per run, never one per process, so two runs never share effects.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Role 2, whoever builds the agent.&lt;/em&gt; Write the contract from your policy, not from a golden run. A ticket with an open carrier exception must end on hold:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;HOLD&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Contract&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tickets&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;T9&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;status&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;on_hold&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="n"&gt;allowed&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tickets&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;T9&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;note&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;tickets&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;T9&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;closed&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)},&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The check reads final values for the required fields, and flags any effect outside both sets. Put it on the closing tool. In the OpenAI Agents SDK, a tool input guardrail runs before the tool executes and can reject the call with a message:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;agents&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;ToolGuardrailFunctionOutput&lt;/span&gt;
&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;agents.decorators&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;tool&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tool_input_guardrail&lt;/span&gt;

&lt;span class="nd"&gt;@tool_input_guardrail&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;records_match_task&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;data&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;problems&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;contract_for&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;current_task&lt;/span&gt;&lt;span class="p"&gt;()).&lt;/span&gt;&lt;span class="nf"&gt;check&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;db&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;effect_log&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;effects&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ToolGuardrailFunctionOutput&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;reject_content&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
            &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;Not closed. Fix these first:&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="se"&gt;\n&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;problems&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;ToolGuardrailFunctionOutput&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;allow&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="nd"&gt;@tool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tool_input_guardrails&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;records_match_task&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;close_ticket&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;ticket_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;-&amp;gt;&lt;/span&gt; &lt;span class="nb"&gt;str&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="bp"&gt;...&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The rejection message goes back to the model as the tool's output. Without that SDK, do the same check at the top of the tool function. Cap the retries, then escalate to a person.&lt;/p&gt;

&lt;p&gt;&lt;em&gt;Role 3, whoever runs the evals.&lt;/em&gt; Run each task several times, each from a fresh copy of the data, and count the tasks that pass every run. The fresh copy matters: a run that starts on rows an earlier run changed can pass for the wrong reason. That number is called pass^k:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;every&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;run_task&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nf"&gt;fresh_db&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;t&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;tasks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;pass^&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;k&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;: &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;every&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;tasks&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;:&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;How you know it worked.&lt;/strong&gt;&lt;/p&gt;

&lt;p&gt;Replay four runs: one right, one with a wrong value, one with an extra write, one with a missing write. A transcript check passes all four. Your gate must close the first and refuse the other three, naming the field each time. In production, count refusals by kind. A rising count of extra effects usually means the agent has a tool it should not have.&lt;/p&gt;

&lt;h2&gt;
  
  
  When a Completion Check Is the Wrong Tool
&lt;/h2&gt;

&lt;p&gt;When you cannot say what the end state should be. A research agent or a drafting agent has no record to check. Review its output instead, or grade it with a rubric.&lt;/p&gt;

&lt;p&gt;When the effect leaves your system. An email, a payment to a third party, or a message in someone else's queue cannot be read back or undone. A check after the fact only tells you what went wrong. Require a person's approval before those calls instead.&lt;/p&gt;

&lt;p&gt;When the task types multiply faster than you can write contracts. Write contracts only for the tasks that change money, status, or anything a customer sees. Let the rest through with an effect log you review weekly.&lt;/p&gt;

&lt;p&gt;When the agent only reads. A read-only agent has no effects. Its failures are wrong answers, and those belong in an answer eval.&lt;/p&gt;

&lt;p&gt;The authors did not measure how much a check like this raises the pass rate, and neither did this session. Measure it on your own tasks before you promise anyone a number.&lt;/p&gt;

&lt;p&gt;Before you add a gate, ask:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Can I write down, for this task type, which fields must change and to what?&lt;/li&gt;
&lt;li&gt;Does every write go through a layer I can log?&lt;/li&gt;
&lt;li&gt;What happens to the customer if the gate refuses and nobody picks it up?&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Glossary
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;transcript&lt;/strong&gt; — the messages and tool results an agent has seen so far in a run.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;effect&lt;/strong&gt; — one write the agent made: table, row, field, and the value before and after.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;contract&lt;/strong&gt; — for one task type, the fields that must end at a value and those it may touch.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;extra effect&lt;/strong&gt; — a write to a field the task's contract does not allow.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;tool input guardrail&lt;/strong&gt; — an Agents SDK check that runs before a tool and can reject the call.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;pass^k&lt;/strong&gt; — the share of tasks an agent passes on every one of k clean runs.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Further reading
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;a href="https://huggingface.co/blog/microsoft/thinkingbox" rel="noopener noreferrer"&gt;The Agent Said It Was Done. The Database Disagreed.&lt;/a&gt; — Microsoft Copilot Studio team, on the Hugging Face blog&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://github.com/microsoft/thinkingbox" rel="noopener noreferrer"&gt;ThinkingBox on GitHub&lt;/a&gt; — Microsoft, MIT license&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://openai.github.io/openai-agents-python/guardrails/" rel="noopener noreferrer"&gt;Guardrails: tool guardrails&lt;/a&gt; — OpenAI, Agents SDK docs&lt;/li&gt;
&lt;li&gt;
&lt;a href="https://sierra.ai/blog/benchmarking-ai-agents" rel="noopener noreferrer"&gt;τ-Bench: Benchmarking AI agents for the real world&lt;/a&gt; — Karthik Narasimhan, Sierra&lt;/li&gt;
&lt;/ul&gt;




&lt;p&gt;&lt;em&gt;Originally published at &lt;a href="https://theaicommit.com/2026-10-19-how-to-make-your-agent-check-the-records-it-changed-before/" rel="noopener noreferrer"&gt;theaicommit.com&lt;/a&gt;, a daily AI engineering lab for software engineers. The original has an &lt;a href="https://theaicommit.com/2026-10-19-how-to-make-your-agent-check-the-records-it-changed-before/" rel="noopener noreferrer"&gt;interactive version&lt;/a&gt; you can play with, and the full runnable code is on &lt;a href="https://github.com/magna56/aI_daily_run/blob/main/2026-10-19/code_example.py" rel="noopener noreferrer"&gt;GitHub&lt;/a&gt;.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>ai</category>
      <category>agents</category>
      <category>testing</category>
      <category>devops</category>
    </item>
  </channel>
</rss>
