<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Mrinmoy Kumar Das</title>
    <description>The latest articles on DEV Community by Mrinmoy Kumar Das (@iammkd).</description>
    <link>https://dev.to/iammkd</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4257228%2Fe7289811-5808-4b38-b11f-23ecb03cc50e.png</url>
      <title>DEV Community: Mrinmoy Kumar Das</title>
      <link>https://dev.to/iammkd</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/iammkd"/>
    <language>en</language>
    <item>
      <title>I chaos-tested my AI agent's LLM provider. 66% of runs failed, and the fix was one argument.</title>
      <dc:creator>Mrinmoy Kumar Das</dc:creator>
      <pubDate>Sat, 21 Nov 2026 11:44:03 +0000</pubDate>
      <link>https://dev.to/iammkd/i-chaos-tested-my-ai-agents-llm-provider-66-of-runs-failed-and-the-fix-was-one-argument-2e95</link>
      <guid>https://dev.to/iammkd/i-chaos-tested-my-ai-agents-llm-provider-66-of-runs-failed-and-the-fix-was-one-argument-2e95</guid>
      <description>&lt;p&gt;My agent passed every demo. It called its tool, answered correctly, and the eval was green.&lt;/p&gt;

&lt;p&gt;Then I asked a different question: &lt;strong&gt;what happens when the model provider misbehaves?&lt;/strong&gt; Rate limits, 503s, a request that just hangs. In production, those aren't edge cases; they're Tuesday.&lt;/p&gt;

&lt;p&gt;So I built &lt;a href="https://github.com/iam-mkd/RunWeave" rel="noopener noreferrer"&gt;RunWeave&lt;/a&gt;, an open-source test harness for AI agents, and pointed it at my own agent. Here's what it found.&lt;/p&gt;

&lt;p&gt;&lt;a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyl6x5h24pab7c7c62g56.gif" class="article-body-image-wrapper"&gt;&lt;img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fyl6x5h24pab7c7c62g56.gif" alt="RunWeave in 12 seconds" width="760" height="451"&gt;&lt;/a&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The agent
&lt;/h2&gt;

&lt;p&gt;A small weather agent: the &lt;code&gt;openai&lt;/code&gt; client, a Chat Completions tool loop, one tool (&lt;code&gt;get_weather&lt;/code&gt;). Nothing exotic; it's &lt;a href="https://github.com/iam-mkd/RunWeave/tree/main/examples/plain-python" rel="noopener noreferrer"&gt;&lt;code&gt;examples/plain-python&lt;/code&gt;&lt;/a&gt; in the repo. The one RunWeave-specific line: the client talks to RunWeave's LLM proxy instead of OpenAI directly.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nc"&gt;AsyncOpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;api_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;proxy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;base_url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;proxy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;openai_base_url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_retries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;By default the proxy answers with a &lt;strong&gt;deterministic mock model&lt;/strong&gt; (scripted by a small fixtures file), so everything below runs offline, for free, with no API key.&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 1: does it give the right answers?
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip install runweave      &lt;span class="c"&gt;# or: uv tool install runweave&lt;/span&gt;
runweave &lt;span class="nb"&gt;eval&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;





&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;plain-python-weather on smoke: 4/4 cases passed (100%), p95 193.2 ms, 703 tokens, $0
Gates passed.
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The dataset checks that the agent calls the tool, answers from its result, and rejects an empty question. Green. Ship it?&lt;/p&gt;

&lt;h2&gt;
  
  
  Step 2: break the provider on purpose
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;runweave bench&lt;/code&gt; load-tests the agent. &lt;code&gt;--fault-profile&lt;/code&gt; makes the proxy misbehave from a fixed seed: the &lt;code&gt;chaos&lt;/code&gt; profile mixes rate limits, 502s, timeouts, slow first tokens, dropped streams, malformed tool calls, unknown tools and context-length errors. The same seed gives the same failures every run, so a failure can be reproduced.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;runweave bench --concurrency 4 --runs 50 --fault-profile &lt;span class="s2"&gt;""&lt;/span&gt; --fault-profile chaos
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Faults&lt;/th&gt;
&lt;th&gt;Runs&lt;/th&gt;
&lt;th&gt;Failed&lt;/th&gt;
&lt;th&gt;p50&lt;/th&gt;
&lt;th&gt;p95&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;none&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;td&gt;50 ms&lt;/td&gt;
&lt;td&gt;69 ms&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;chaos&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;33&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;30.0 s&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;30.0 s&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two thirds of the runs failed, and every failure was the same: &lt;code&gt;TIMED_OUT: RUN_DEADLINE_EXCEEDED&lt;/code&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What actually broke
&lt;/h2&gt;

&lt;p&gt;Rate limits and 502s weren't the problem. The &lt;code&gt;openai&lt;/code&gt; client already retries those (&lt;code&gt;max_retries=2&lt;/code&gt;). The problem was the &lt;strong&gt;hang&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;When the provider stalls, the &lt;code&gt;openai&lt;/code&gt; client waits up to its default timeout, &lt;strong&gt;10 minutes per request&lt;/strong&gt;. My run's deadline is 30 seconds. So one stalled call ate the entire budget, and the retries never got a chance to run.&lt;/p&gt;

&lt;p&gt;In production, that's a user staring at a spinner for 30 seconds, then an error, while my retry logic sat there unused.&lt;/p&gt;

&lt;h2&gt;
  
  
  The fix: one argument
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;async&lt;/span&gt; &lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nc"&gt;AsyncOpenAI&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="n"&gt;api_key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;proxy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;base_url&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;ctx&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;proxy&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;openai_base_url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="n"&gt;max_retries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mf"&gt;5.0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Same command, same seed:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Faults&lt;/th&gt;
&lt;th&gt;Runs&lt;/th&gt;
&lt;th&gt;Failed&lt;/th&gt;
&lt;th&gt;p50&lt;/th&gt;
&lt;th&gt;p95&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;chaos&lt;/td&gt;
&lt;td&gt;50&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;0&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;5.5 s&lt;/td&gt;
&lt;td&gt;5.6 s&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Zero failures. A stalled call now gives up after 5 seconds and the retry succeeds within the run's deadline. Latency under chaos is still high, but bounded, and the agent recovers instead of timing out.&lt;/p&gt;

&lt;p&gt;The general rule: &lt;strong&gt;the client timeout times the number of attempts must fit inside your run's deadline.&lt;/strong&gt; Obvious in hindsight. I'd never have found it without injecting the fault.&lt;/p&gt;

&lt;h2&gt;
  
  
  Put it in CI
&lt;/h2&gt;

&lt;p&gt;&lt;code&gt;runweave eval&lt;/code&gt; exits 1 when a gate fails, and writes JUnit, so a regression blocks the pull request. Bench gates work too:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight toml"&gt;&lt;code&gt;&lt;span class="k"&gt;[bench]&lt;/span&gt;
&lt;span class="na"&gt;concurrency&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="na"&gt;runs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;50&lt;/span&gt;
&lt;span class="na"&gt;fault_profiles&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;""&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"chaos"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="na"&gt;max_failure_rate&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;      &lt;span class="c1"&gt;# any failed run fails the job&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;There's a copy-ready GitHub Actions job in &lt;a href="https://github.com/iam-mkd/RunWeave/tree/main/examples/github-actions" rel="noopener noreferrer"&gt;&lt;code&gt;examples/github-actions&lt;/code&gt;&lt;/a&gt;.&lt;/p&gt;

&lt;h2&gt;
  
  
  What else is in there
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;code&gt;runweave shadow&lt;/code&gt;: run the old and new version on the same cases and see which gates the new one passes.&lt;/li&gt;
&lt;li&gt;A local workspace (&lt;code&gt;runweave serve&lt;/code&gt;) with a live timeline of every model call (with tokens) and tool call.&lt;/li&gt;
&lt;li&gt;Examples for plain &lt;code&gt;openai&lt;/code&gt;, the OpenAI Agents SDK, the Claude Agent SDK, LangGraph and TypeScript (&lt;code&gt;npm install runweave-agent&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Your own key, when you're ready for real models, with a spend cap per run.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Honest limits
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;A mock benchmark measures &lt;strong&gt;your agent and RunWeave on your machine&lt;/strong&gt;, not provider capacity. Real providers have their own limits.&lt;/li&gt;
&lt;li&gt;It's local-only (binds to 127.0.0.1) and v0.1.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  Try it on your agent
&lt;/h2&gt;



&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip install runweave
runweave init my-agent &lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt; &lt;span class="nb"&gt;cd&lt;/span&gt; my-agent
runweave &lt;span class="nb"&gt;eval&lt;/span&gt;
runweave bench --fault-profile &lt;span class="s2"&gt;""&lt;/span&gt; --fault-profile chaos
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;strong&gt;I'm curious: what's your agent's failure rate under &lt;code&gt;chaos&lt;/code&gt;?&lt;/strong&gt; And what's the weirdest way an agent of yours has failed in production? I'd like to add it as a fault.&lt;/p&gt;

&lt;p&gt;⭐ &lt;a href="https://github.com/iam-mkd/RunWeave" rel="noopener noreferrer"&gt;github.com/iam-mkd/RunWeave&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;Note: at the end of a chaos run you may see CancelledError … graceful shutdown exceeded tracebacks. They come from requests the timeout fault was still holding open after the agent had moved on. They're harmless, and the result above is unaffected; a fix is coming in the next release.&lt;/p&gt;

</description>
      <category>python</category>
      <category>ai</category>
      <category>testing</category>
      <category>opensource</category>
    </item>
  </channel>
</rss>
