<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Shalendra gaur</title>
    <description>The latest articles on DEV Community by Shalendra gaur (@shalendra_gaur_ecd0567a49).</description>
    <link>https://dev.to/shalendra_gaur_ecd0567a49</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4239346%2F6b7c2806-2fb0-4c49-bca3-3ab251f3ace1.png</url>
      <title>DEV Community: Shalendra gaur</title>
      <link>https://dev.to/shalendra_gaur_ecd0567a49</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/shalendra_gaur_ecd0567a49"/>
    <language>en</language>
    <item>
      <title>A 98%-accurate model leaves an agent's state right only 21% of the time without AXI</title>
      <dc:creator>Shalendra gaur</dc:creator>
      <pubDate>Sat, 14 Nov 2026 11:39:20 +0000</pubDate>
      <link>https://dev.to/shalendra_gaur_ecd0567a49/a-98-accurate-model-leaves-an-agents-state-right-only-21-of-the-time-without-axi-2c49</link>
      <guid>https://dev.to/shalendra_gaur_ecd0567a49/a-98-accurate-model-leaves-an-agents-state-right-only-21-of-the-time-without-axi-2c49</guid>
      <description>&lt;p&gt;Measured on Gemini 3.8 Flash over 4,000 requests. Every number below comes from files in a public repo that you can replay.&lt;br&gt;
Disclosure: the architecture behind AXI is Rajnish Choubey's. I built and ran the tests. An AI assistant helped write the test code and this article.&lt;br&gt;
In short&lt;br&gt;
• On a 4,000-request job, the model's reply was right 3,954 times (98.85%).&lt;br&gt;
• Applied as written, those same replies left the agent's state exactly right on only 85 of the first 400 steps (21%).&lt;br&gt;
• With AXI between the model and the state, the state was exactly right on 4,000 of 4,000 steps.&lt;br&gt;
• Scope: one model, one job, one seed. The claims are about the state the host holds. The model is unchanged.&lt;br&gt;
Why a 98% model can leave the state right 21% of the time&lt;br&gt;
A model that is right 98.85% of the time is wrong about once in 87 replies. In a loop that applies each reply as it comes, one wrong reply is not one error. It is a changed state, and every later step builds on it.&lt;br&gt;
Here is the kind of slip that did the damage. Asked to delete a service and also remove one of its edges, the model removed those, but left out the service's other connections. The service was gone and edges still pointed at it. All 46 slips in the run were of this one kind. On everything else it was asked, the model was right.&lt;br&gt;
Applied as written, a slip like that stayed in the state for 36 to 150 steps at a time, until a later request happened to clear it. The last one was still going when the 400 steps ended.&lt;br&gt;
What AXI is&lt;br&gt;
AXI is a layer on the host side, between a model and the state it edits. The model proposes one change at a time. AXI checks the change against the rules, completes what the rules can complete, commits it all or not at all, and then shows the model the exact state again. The model is unchanged. The claims here are about the state the host holds. It is meant for the companies that run models and agent loops.&lt;br&gt;
The idea, the framework and the architecture are Rajnish Choubey's. I built the tests and ran them.&lt;br&gt;
The test&lt;br&gt;
• Model: Gemini 3.8 Flash, default settings.&lt;br&gt;
• Job: editing a service graph of 17 to 22 services on average, under constant change.&lt;br&gt;
• Size: 4,000 requests in a row, 3.7 million tokens sent.&lt;br&gt;
• Before the run, the plan, the reading rules and my predictions (with a probability on each) were written and hash-locked. I rated the chance that all nine headline rules would pass at about 20%. They did. One other rule I had set failed: the bar was that applying the replies as written would corrupt at least 80% of the first 400 steps, and it was 55.8%.&lt;br&gt;
Results&lt;br&gt;
First 400 requests of the same stream, three ways of keeping the state:&lt;/p&gt;

&lt;p&gt;steps with the right state (of 400)&lt;br&gt;
steps with a corrupted state&lt;br&gt;
cost of one right step&lt;br&gt;
AXI (repair + scope filter)&lt;br&gt;
400&lt;br&gt;
0&lt;br&gt;
$0.0012&lt;br&gt;
Replies applied as written&lt;br&gt;
85 (21%)&lt;br&gt;
223&lt;br&gt;
$0.0052 (4.5x)&lt;br&gt;
A gate that only refuses bad replies&lt;br&gt;
84&lt;br&gt;
0&lt;br&gt;
$0.0057 (4.9x)&lt;br&gt;
• The whole run with AXI: exactly right on 4,000 of 4,000 steps, never corrupted. The prompt stayed about the same size (1.16 times from the first tenth of the run to the last) and the speed did not fall (0.94 times).&lt;br&gt;
• The model itself: 3,954 of 4,000 replies right. A fixed rule completed all 46 slips.&lt;br&gt;
• A gate that only refuses bad replies never held a corrupted state, but it was off the right state on every step from 210 to 400. Refusing a bad reply keeps the state well formed. It does not make it right.&lt;br&gt;
The three ways spent about the same: $0.47, $0.45 and $0.48. What differed is how many steps ended with the right state, so what one right step cost differed too.&lt;br&gt;
Against a chat that keeps its whole history&lt;br&gt;
A fair question is how this compares with simply keeping the conversation. A separate test, with 40 lines of operations log added to every request, ran both on the same job.&lt;br&gt;
• With AXI: exactly right on 1,500 of 1,500 steps, at about 2,800 prompt tokens and 3.07 seconds a step.&lt;br&gt;
• The kept chat was exact for its first 222 turns, then slipped 15 times before the run stopped at 457 turns. Applied as each reply came, its held state was wrong on every turn from turn 287 on. A call at a prompt of 500,000 tokens or more took 3.5 times as long as a step with AXI.&lt;br&gt;
That chat was never shortened. Chats that are shortened by summaries are a different test, and I will publish that one separately with the raw files.&lt;br&gt;
What this does not show&lt;br&gt;
• One model, one job and one seed per run. The second seed of the 4,000-request run has not been made.&lt;br&gt;
• The repair rule (a deleted service takes its dependants with it) is the job's own rule, and the model was never told it. This shows the layer works when the rule is known. It does not show that it can find rules.&lt;br&gt;
• The applied-as-written and gate arms ran for the first 400 requests only, and they share replies with each other, so they are not three independent samples.&lt;br&gt;
• The cost per right step is arithmetic on the saved files at the prices I typed in ($0.75 per million tokens in, $3.75 out, thinking counted as out). I added it after the results were read, so it is not one of the pre-written rules.&lt;br&gt;
• Nothing here is about another model, another task, or energy.&lt;br&gt;
Check it yourself&lt;br&gt;
The plans, the saved replies and the tests are in the public repo: &lt;a href="https://github.com/gaurshalendra2412/axi-runtime" rel="noopener noreferrer"&gt;https://github.com/gaurshalendra2412/axi-runtime&lt;/a&gt;. The pages are docs/MILLION_TOKEN_TEST_HARD.md (the 4,000-request run) and docs/GEMINI_LONG_TEST.md (the kept chat). Every saved reply is there, and the tests replay them to reproduce the numbers above.&lt;br&gt;
White paper: [&lt;a href="https://zenodo.org/records/23958737" rel="noopener noreferrer"&gt;https://zenodo.org/records/23958737&lt;/a&gt;]&lt;br&gt;
AXI is Rajnish Choubey's architecture. Tests and write-up: Shalendra Gaur.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>agents</category>
      <category>infrastructure</category>
    </item>
    <item>
      <title>A model right 98% of the time left the state right on only 21% of steps</title>
      <dc:creator>Shalendra gaur</dc:creator>
      <pubDate>Thu, 12 Nov 2026 09:52:17 +0000</pubDate>
      <link>https://dev.to/shalendra_gaur_ecd0567a49/a-model-right-98-of-the-time-left-the-state-right-on-only-21-of-steps-2jh8</link>
      <guid>https://dev.to/shalendra_gaur_ecd0567a49/a-model-right-98-of-the-time-left-the-state-right-on-only-21-of-steps-2jh8</guid>
      <description>&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faqw2huwoqw2rrtpgcj4k.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Faqw2huwoqw2rrtpgcj4k.gif" alt=" " width="600" height="338"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;We ran an AI agent on a service graph for 400 requests (the first 400 of a 4,000-request run on Gemini 3.8 Flash). 393 of the 400 replies had the right items. But when the replies were applied as written, the state was exactly right on only 85 of 400 steps (21%). With AXI it was exactly right on 400 of 400.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why 98% right becomes 21% right
&lt;/h2&gt;

&lt;p&gt;A slip that nothing catches stays in the state, and every later step builds on it. Applied as written, the state was off the right one on 315 of the 400 steps. A small error rate per reply becomes a large error rate per job.&lt;/p&gt;

&lt;h2&gt;
  
  
  What AXI does
&lt;/h2&gt;

&lt;p&gt;AXI is an architecture by &lt;strong&gt;Rajnish Choubey&lt;/strong&gt;. The host holds the exact state of the job. At every step:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The host builds a short prompt from the exact state.&lt;/li&gt;
&lt;li&gt;The model proposes the next change.&lt;/li&gt;
&lt;li&gt;A deterministic gate checks it (about 2 microseconds, measured in Python).&lt;/li&gt;
&lt;li&gt;A rule-based repair fixes a slip.&lt;/li&gt;
&lt;li&gt;The host updates the state.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;There is no training and no change to the model. It works from the first step.&lt;/p&gt;

&lt;h2&gt;
  
  
  What it costs
&lt;/h2&gt;

&lt;p&gt;On the same 400 requests, one step with the right state cost $0.0012 with AXI and $0.0052 when the replies were applied as written (4.5 times). The total spend was about the same ($0.45 to $0.47). The saving is in the cost of a right step, not in a lower bill.&lt;/p&gt;

&lt;h2&gt;
  
  
  A longer run
&lt;/h2&gt;

&lt;p&gt;In a separate 1,500-step test on Gemini 3.8 Flash, the host-held state was exact on all 1,500 steps, with a prompt of about 2,824 tokens at every step. A chat that kept its whole history had a mean prompt of 476,835 tokens (we stopped it at 457 turns). It was exact for its first 222 turns, then slipped, and applied as written its state was wrong on every turn from turn 287 on.&lt;/p&gt;

&lt;h2&gt;
  
  
  Don't take our word for it
&lt;/h2&gt;

&lt;p&gt;The test pages were locked with a SHA-256 hash before the runs. A test in the repo recomputes every figure from the saved results. There is a Colab notebook that runs it on your own Gemini key with a spend limit.&lt;/p&gt;

&lt;p&gt;What this does not show: it is one run on one model, and the cost figure is from the first 400 requests. A head-to-head comparison against summarisation and other caching set-ups has not been run yet. If you find a number that is wrong, please tell me.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Repo: &lt;a href="https://github.com/gaurshalendra2412/axi-runtime" rel="noopener noreferrer"&gt;https://github.com/gaurshalendra2412/axi-runtime&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Paper: &lt;a href="https://doi.org/10.5281/zenodo.23958737" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23958737&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Tests built and run by Shalendra Gaur, published with Rajnish Choubey's permission.&lt;/p&gt;

&lt;h1&gt;
  
  
  ai #llm #aiagents #tokencost
&lt;/h1&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>architecture</category>
    </item>
    <item>
      <title>LLM inference cost: the quadratic term nobody removes, and a measured fix</title>
      <dc:creator>Shalendra gaur</dc:creator>
      <pubDate>Thu, 12 Nov 2026 09:18:57 +0000</pubDate>
      <link>https://dev.to/shalendra_gaur_ecd0567a49/llm-inference-cost-the-quadratic-term-nobody-removes-and-a-measured-fix-23a8</link>
      <guid>https://dev.to/shalendra_gaur_ecd0567a49/llm-inference-cost-the-quadratic-term-nobody-removes-and-a-measured-fix-23a8</guid>
      <description>&lt;p&gt;Everyone is talking about LLM inference cost. The price per token keeps falling, yet bills keep rising, especially for AI agents. This article explains one cause that gets little attention, shows the arithmetic, and reports what happened when we removed it in a measured test on Gemini 3.8 Flash.&lt;/p&gt;

&lt;h2&gt;
  
  
  The short version
&lt;/h2&gt;

&lt;p&gt;An agent that keeps its history in the prompt pays for the same tokens again at every step. The total grows with the &lt;strong&gt;square&lt;/strong&gt; of the number of steps. Cheaper chips and cheaper tokens make each token cost less, but they don't change that curve. Holding the exact state of the job on the host and sending a fresh short prompt at every step makes the cost grow in a &lt;strong&gt;straight line&lt;/strong&gt; instead.&lt;/p&gt;

&lt;h2&gt;
  
  
  Where the cost comes from
&lt;/h2&gt;

&lt;p&gt;Most agents work like a chat. At every step the model receives the whole history again: the instructions, every earlier request, every earlier answer. Step &lt;em&gt;k&lt;/em&gt; sends about &lt;em&gt;k&lt;/em&gt; units of history. Add up &lt;em&gt;N&lt;/em&gt; steps and you get about N²/2 units. Double the length of a job and the history cost quadruples.&lt;/p&gt;

&lt;p&gt;This has a consequence people feel but rarely price: &lt;strong&gt;the more an agent is used, the worse the cost per task gets.&lt;/strong&gt; Under a flat-price plan, the heaviest users cost the most. Under per-token pricing, the same job costs more than it did the week before simply because it ran longer.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the cure has to change
&lt;/h2&gt;

&lt;p&gt;There are three common ways to reduce the bill, and it helps to see what each one changes:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Cheaper hardware or cheaper tokens&lt;/strong&gt; reduce the price of each token by a constant factor. The curve keeps its shape.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Prefix caching&lt;/strong&gt; makes re-reading tokens cheaper by a fixed factor. Again, a constant, and the curve keeps its shape.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Summaries and compaction&lt;/strong&gt; shrink the history, but a summary is a guess about what matters, and a lost detail stays lost.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;To change the shape, the history itself has to stop growing.&lt;/p&gt;

&lt;h2&gt;
  
  
  The approach we tested
&lt;/h2&gt;

&lt;p&gt;The architecture, &lt;strong&gt;AXI&lt;/strong&gt;, is by &lt;strong&gt;Rajnish Choubey&lt;/strong&gt;. The exact state of the job lives on the host. At every step:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;The host builds a short prompt from the exact state.&lt;/li&gt;
&lt;li&gt;The model proposes the next move.&lt;/li&gt;
&lt;li&gt;A deterministic gate checks the proposal in about 2 microseconds (measured in Python).&lt;/li&gt;
&lt;li&gt;A rule-based repair fixes a slip.&lt;/li&gt;
&lt;li&gt;The host updates the state and the loop repeats.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;There is no training and no change to the model. It works from the first step, on short jobs and long ones.&lt;/p&gt;

&lt;h2&gt;
  
  
  A worked example from the measurements
&lt;/h2&gt;

&lt;p&gt;Over 1,500 steps on Gemini 3.8 Flash, the kept chat's mean prompt was &lt;strong&gt;476,835 tokens&lt;/strong&gt; (peak 949,591). With host-held state, every prompt was &lt;strong&gt;2,824 tokens&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;Total input over the job, from those means:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Kept chat: 476,835 × 1,500 ≈ &lt;strong&gt;715 million tokens&lt;/strong&gt;
&lt;/li&gt;
&lt;li&gt;Host-held state: 2,824 × 1,500 ≈ &lt;strong&gt;4.2 million tokens&lt;/strong&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;That is about 169 times fewer input tokens.&lt;/p&gt;

&lt;p&gt;The cost figure is smaller than that token ratio, because it counts every call at typed prices per &lt;strong&gt;correct&lt;/strong&gt; step: &lt;strong&gt;$0.0012 against $0.0052, 4.5 times lower&lt;/strong&gt;, and 9.3 times lower on a second seed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Latency, not just money
&lt;/h2&gt;

&lt;p&gt;Past 500k tokens of context, a whole call took &lt;strong&gt;10.66 s against 3.07 s&lt;/strong&gt; (3.5x), and the first token arrived after &lt;strong&gt;9.13 s against 2.27 s&lt;/strong&gt; (4.0x). Exact steps per hour: &lt;strong&gt;1,280 against 488&lt;/strong&gt; (2.6x). A smaller prompt is cheaper and also faster.&lt;/p&gt;

&lt;h2&gt;
  
  
  Accuracy is part of cost
&lt;/h2&gt;

&lt;p&gt;Wrong steps cost money twice: once to make them and again to fix them. In the same 1,500-step run, the host-held approach was exact on 1,500 of 1,500 steps, while the kept chat started slipping at turn 223 and was wrong from turn 287.&lt;/p&gt;

&lt;p&gt;In a separate 4,000-request run, 393 of 400 replies were right (98%), but only 85 of 400 steps ended in the correct state (21%) when the replies were applied as written. A small error rate per reply becomes a large error rate per job, because each slip is carried into everything after it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Questions people ask
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Doesn't prefix caching already solve this?&lt;/strong&gt;&lt;br&gt;
Caching cuts the price of re-reading, by a fixed factor. It doesn't stop the history from growing, doesn't remove the latency we measured past 500k tokens, and doesn't stop drift. A direct head-to-head comparison on the same job is the right next experiment, and I would welcome help running it.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Does it need fine-tuning or a special model?&lt;/strong&gt;&lt;br&gt;
No. The model is unchanged. We tested Gemini 3.8 Flash and small Qwen models. With Qwen, 12 of 12 kept-chat episodes ended corrupted and 0 of 360 did with host-held state.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Is this only for long jobs?&lt;/strong&gt;&lt;br&gt;
No. It works from the first step. The saving grows with the number of steps, but the exactness holds from the start.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;How do I know the numbers are real?&lt;/strong&gt;&lt;br&gt;
Each test page was locked with a SHA-256 hash before the run, with probabilities on every prediction. In the latest test, 12 of 13 predictions held and the one that missed is reported as wrong. A test in the repo recomputes every figure from the saved results.&lt;/p&gt;

&lt;h2&gt;
  
  
  Compare it on your own workload
&lt;/h2&gt;

&lt;p&gt;If you run agents at scale, the quickest way to see what this would save you is to try it on your own jobs. The repo has a Colab notebook that runs the test with your own Gemini key and a spend limit. If you try it, please open an issue with what you find, good or bad.&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Repo with tests: &lt;a href="https://github.com/gaurshalendra2412/axi-runtime" rel="noopener noreferrer"&gt;https://github.com/gaurshalendra2412/axi-runtime&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Paper: &lt;a href="https://doi.org/10.5281/zenodo.23958737" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23958737&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Tests built and run by Shalendra Gaur, published with Rajnish Choubey's permission.&lt;/p&gt;

&lt;h1&gt;
  
  
  inference #llm #ai #aiagents #tokencost #kvcache #contextwindow #prefixcaching #machinelearning #devops #mlops #gemini #opensource #performance
&lt;/h1&gt;

</description>
      <category>ai</category>
      <category>machinelearning</category>
      <category>architecture</category>
      <category>infrastructure</category>
    </item>
    <item>
      <title>Your LLM agent re-reads its history every step. We measured what that costs</title>
      <dc:creator>Shalendra gaur</dc:creator>
      <pubDate>Wed, 11 Nov 2026 15:16:03 +0000</pubDate>
      <link>https://dev.to/shalendra_gaur_ecd0567a49/your-llm-agent-re-reads-its-history-every-step-we-measured-what-that-costs-jjg</link>
      <guid>https://dev.to/shalendra_gaur_ecd0567a49/your-llm-agent-re-reads-its-history-every-step-we-measured-what-that-costs-jjg</guid>
      <description>&lt;p&gt;At turn 223, a Gemini 3.8 Flash chat started making mistakes. By turn 287 it was wrong. The same model, on the same job, with the same 1,500 steps, made none.&lt;/p&gt;

&lt;p&gt;The difference was not the model. It was what the model was handed at each step.&lt;/p&gt;

&lt;h2&gt;
  
  
  The cost nobody prices in
&lt;/h2&gt;

&lt;p&gt;Most LLM agents work like a chat: every step, the whole history goes back into the prompt. Step &lt;em&gt;k&lt;/em&gt; sends about &lt;em&gt;k&lt;/em&gt; units of history, so a job of &lt;em&gt;N&lt;/em&gt; steps sends about N²/2 units in total. Double the job and the cost of the history quadruples. Faster chips make each token cheaper by a constant factor. They don't change that curve.&lt;/p&gt;

&lt;p&gt;We measured it. Over 1,500 steps the kept chat's mean prompt was &lt;strong&gt;476,835 tokens&lt;/strong&gt;, peaking at &lt;strong&gt;949,591&lt;/strong&gt;. The alternative held the exact state of the job on the host and sent a fresh prompt of &lt;strong&gt;2,824 tokens&lt;/strong&gt; at every step. That is &lt;strong&gt;169 times smaller&lt;/strong&gt; on average.&lt;/p&gt;

&lt;h2&gt;
  
  
  What that did
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Exactness:&lt;/strong&gt; 1,500 of 1,500 steps exact, against a chat that slipped from turn 223.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Latency:&lt;/strong&gt; past 500k tokens, a call took 10.66 s against 3.07 s, and first token 9.13 s against 2.27 s.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Throughput:&lt;/strong&gt; 1,280 exact steps per hour against 488 (2.6x).&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Cost per correct step&lt;/strong&gt; at typed prices: $0.0012 against $0.0052 (4.5x), and 9.3x on a second seed.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  The error problem is worse than the cost problem
&lt;/h2&gt;

&lt;p&gt;In a separate 4,000-request run, 393 of 400 replies were right, 98%. Applied exactly as the model wrote them, only 85 of 400 steps ended in the correct state, 21%. A small error rate per reply becomes a large error rate per job, because each slip is carried into everything after it.&lt;/p&gt;

&lt;p&gt;In a fuzzy-input test, one slip at turn 191 stayed in the state for 140 turns, while 99.5% of replies were individually fine.&lt;/p&gt;

&lt;p&gt;On a small open model (Qwen), 12 of 12 episodes with the history kept in the chat ended corrupted. With host-held state, 0 of 360 did.&lt;/p&gt;

&lt;h2&gt;
  
  
  How it works
&lt;/h2&gt;

&lt;p&gt;The architecture, &lt;strong&gt;AXI&lt;/strong&gt;, is by Rajnish Choubey. The exact state lives on the host. At each step the model gets a short prompt built from it, and proposes the next move. A deterministic gate checks the proposal in about 2 microseconds (measured in Python), a rule-based repair fixes a slip, and the host updates the state.&lt;/p&gt;

&lt;p&gt;There is no training and no change to the model, and it works from the first step, on short jobs and long ones. The cost grows linearly with steps, not with their square.&lt;/p&gt;

&lt;h2&gt;
  
  
  How we tested
&lt;/h2&gt;

&lt;p&gt;Every test page was locked with a SHA-256 hash before the run, with probabilities attached to each prediction. In the latest test, 12 of 13 predictions held and one missed; we report it as wrong. A test in the repo recomputes every figure from the saved results.&lt;/p&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;These are specific jobs on Gemini 3.8 Flash and Qwen.On a TPU v6e-1, memory headroom allowed 927 sessions with host-held state against 5 with a kept chat, but that is for a 0.5B cache shape and is not end-to-end throughput.&lt;/p&gt;

&lt;p&gt;Tests built and run by Shalendra Gaur, published with Rajnish Choubey's permission.&lt;/p&gt;

&lt;h2&gt;
  
  
  Try it yourself
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Repo with tests and a Colab notebook (your own Gemini key, with a spend limit): &lt;a href="https://github.com/gaurshalendra2412/axi-runtime" rel="noopener noreferrer"&gt;https://github.com/gaurshalendra2412/axi-runtime&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Paper: &lt;a href="https://doi.org/10.5281/zenodo.23958737" rel="noopener noreferrer"&gt;https://doi.org/10.5281/zenodo.23958737&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;h1&gt;
  
  
  ai #llm #machinelearning #python #gemini #aiagents #tokencost #kvcache #inference #devops
&lt;/h1&gt;

</description>
      <category>ai</category>
      <category>llm</category>
      <category>machinelearning</category>
      <category>aicost</category>
    </item>
  </channel>
</rss>
