<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: salahmakboul</title>
    <description>The latest articles on DEV Community by salahmakboul (@salahmakboul).</description>
    <link>https://dev.to/salahmakboul</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4217802%2F7823f043-fcc9-4cea-9902-8b0435f9720a.jpg</url>
      <title>DEV Community: salahmakboul</title>
      <link>https://dev.to/salahmakboul</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/salahmakboul"/>
    <language>en</language>
    <item>
      <title>One approval, two payments: what an AI agent does after a crash</title>
      <dc:creator>salahmakboul</dc:creator>
      <pubDate>Sat, 07 Nov 2026 00:20:12 +0000</pubDate>
      <link>https://dev.to/salahmakboul/one-approval-two-payments-what-an-ai-agent-does-after-a-crash-15mp</link>
      <guid>https://dev.to/salahmakboul/one-approval-two-payments-what-an-ai-agent-does-after-a-crash-15mp</guid>
      <description>&lt;p&gt;&lt;em&gt;Written with help from AI (Claude). I maintain Call4Agents, the library mentioned at the end. Every number here&lt;br&gt;
comes from scripts you can rerun, and the fixes in the "What to do" list work without my library.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;A person approves one payment. The tool pays. The process dies before the framework saves its progress. A new&lt;br&gt;
process resumes the way the docs say. The payment runs again.&lt;/p&gt;

&lt;p&gt;I measured this in LangGraph, Pydantic AI and the OpenAI Agents SDK. In all three, one approval gave &lt;strong&gt;2 payments,&lt;br&gt;
in every run&lt;/strong&gt;: 30 runs per framework between 21 October and 5 November, on the newest versions each day. A guard&lt;br&gt;
that records the action outside the framework brought it to &lt;strong&gt;1 payment, every run&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;This isn't a framework bug. It's at-least-once execution, which is what any system gives you when it saves progress&lt;br&gt;
after doing the work. LangGraph's docs already warn about it. OpenAI added a docs section after I asked. Pydantic AI's&lt;br&gt;
triage reproduced it.&lt;/p&gt;

&lt;p&gt;The trouble is the gap between what "approved" sounds like and what it promises. People read "a human approved this"&lt;br&gt;
as "this happens once, exactly as approved". Neither part is guaranteed.&lt;/p&gt;
&lt;h2&gt;
  
  
  The experiment
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Fake models, no network, no API keys. Each step is a separate OS process.&lt;/li&gt;
&lt;li&gt;The crash is &lt;code&gt;os._exit&lt;/code&gt;, so no cleanup code runs. A "payment" is a line added to a local file.&lt;/li&gt;
&lt;li&gt;Steps: the agent asks for &lt;code&gt;pay(amount=10, to="alice")&lt;/code&gt;, which needs approval. A person approves, the tool pays,
and the process dies right after. A new process resumes from the saved state. Count the payments.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;A second case: the saved call is changed to &lt;code&gt;pay(9999, "mallory")&lt;/code&gt; after the pause and before it runs. That models&lt;br&gt;
anyone who can write to the state store.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Framework&lt;/th&gt;
&lt;th&gt;Crash after the approved payment&lt;/th&gt;
&lt;th&gt;Saved call edited after the yes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;LangGraph&lt;/td&gt;
&lt;td&gt;2 payments&lt;/td&gt;
&lt;td&gt;paid 9999 to mallory&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Pydantic AI&lt;/td&gt;
&lt;td&gt;2 payments&lt;/td&gt;
&lt;td&gt;paid 9999 to mallory&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;OpenAI Agents SDK&lt;/td&gt;
&lt;td&gt;2 payments&lt;/td&gt;
&lt;td&gt;refused by the SDK itself&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Any of them, with a guard outside&lt;/td&gt;
&lt;td&gt;1 payment&lt;/td&gt;
&lt;td&gt;nothing paid&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;Two details I don't want to hide:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;The OpenAI SDK deserves credit.&lt;/strong&gt; It refuses changed arguments under the same call id on its own.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;LangGraph sometimes asks for approval again&lt;/strong&gt;, and I can't predict when. When it asks, the second payment
follows a second "yes" from someone who may not know the first one went through. It paid twice either way.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2&gt;
  
  
  Try it in five seconds
&lt;/h2&gt;


&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;pip &lt;span class="nb"&gt;install &lt;/span&gt;call4agents
call4agents incident
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;


&lt;p&gt;This replays three failures, without and with the guard. My run on 6 November took 5.5 seconds:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;Incident 1. The double payment
  A person approves pay(10, alice). The bank charges. The process dies before saving. It resumes.
  without Call4Agents    charges: 2  paid: 10 to alice, 10 to alice
  with Call4Agents       charges: 1  paid: 10 to alice
  The same, but the process dies inside the call, so nobody knows if it finished:
  without Call4Agents    charges: 2  paid: 10 to alice, 10 to alice
  with Call4Agents       charges: 1  paid: 10 to alice (stopped: in doubt, a person must check)

Incident 2. The changed call
  A person approves pay(10, alice). The saved call is changed to pay(9999, mallory) before it runs.
  without Call4Agents    charges: 1  paid: 9999 to mallory
  with Call4Agents       charges: 0  paid: nothing (refused: no yes for this exact call)

Incident 3. The ignored no
  A person says no to pay(10, alice). A bug in the agent loop runs the call anyway.
  without Call4Agents    charges: 1  paid: 10 to alice
  with Call4Agents       charges: 0  paid: nothing (refused: no yes for this exact call)
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The processes and crashes are real. The bank and the agent loop are small local stand-ins, not the frameworks.&lt;/p&gt;

&lt;p&gt;Look at the second line of Incident 1. When the process dies &lt;em&gt;inside&lt;/em&gt; the call, nobody knows whether the payment&lt;br&gt;
went through. The guard doesn't guess and doesn't run it again: it stops as "in doubt". Then a provider check, an&lt;br&gt;
idempotency key or a person decides. I think that's the honest answer, because no library can know what happened in&lt;br&gt;
someone else's bank.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why this happens
&lt;/h2&gt;

&lt;p&gt;Doing the work and saving that you did it are two separate steps, and no process can make them one. If the crash&lt;br&gt;
falls between them, the saved state still says "not done", so a resume does it again.&lt;/p&gt;

&lt;p&gt;Approval adds another trap. Approval state is usually a flag next to the call: &lt;em&gt;this call id was approved&lt;/em&gt;. Nothing&lt;br&gt;
ties the "yes" to the exact arguments, to one use, or to a real pause.&lt;/p&gt;

&lt;h2&gt;
  
  
  My own bug
&lt;/h2&gt;

&lt;p&gt;The library had this kind of bug too. A reader found that in every release from 0.1.0 to 0.5.0, a "yes" for 10 could&lt;br&gt;
let 20 run, with only a crash at the wrong moment. All my checks passed the whole time. It's fixed in 0.5.2, with a&lt;br&gt;
&lt;a href="https://github.com/salahmakboul/call4agents/security/advisories/GHSA-gxg5-95g9-xw4w" rel="noopener noreferrer"&gt;public advisory&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;After that I added a test that kills a real process before and after every write, 412 crashes in total. With the old&lt;br&gt;
bug put back in, that test failed by itself. It would have found my bug before the reader did.&lt;/p&gt;

&lt;h2&gt;
  
  
  What to do, with or without my library
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Use an idempotency key the provider enforces&lt;/strong&gt;, stable across resumes and scoped to your operation.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat "maybe happened" as unknown.&lt;/strong&gt; Without a provider key, don't replay automatically. A read that finds
nothing can lag.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Check the approval at the tool&lt;/strong&gt;, against the arguments about to run, not only at the resume.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Crash your own agent on purpose.&lt;/strong&gt; Kill the process between the action and the save. It takes minutes.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  Limits
&lt;/h2&gt;

&lt;p&gt;Fake models, one machine, no real payment provider, no production traffic. Three frameworks at a few versions,&lt;br&gt;
with the crash at one point. It's a small experiment you can repeat, not a production study.&lt;/p&gt;

&lt;p&gt;The full write-up, with the script and every run's output, is here:&lt;br&gt;
&lt;a href="https://github.com/salahmakboul/call4agents/tree/main/docs/finding" rel="noopener noreferrer"&gt;https://github.com/salahmakboul/call4agents/tree/main/docs/finding&lt;/a&gt;&lt;/p&gt;

&lt;p&gt;If you run an approval flow, try to break it the same way and tell me what happens, with or without my library.&lt;/p&gt;

</description>
      <category>ai</category>
      <category>python</category>
      <category>agents</category>
      <category>testing</category>
    </item>
    <item>
      <title>My approval library let a yes for 10 pay 20. Here is how one crash did it</title>
      <dc:creator>salahmakboul</dc:creator>
      <pubDate>Fri, 30 Oct 2026 21:57:47 +0000</pubDate>
      <link>https://dev.to/salahmakboul/my-approval-library-let-a-yes-for-10-pay-20-here-is-how-one-crash-did-it-34p</link>
      <guid>https://dev.to/salahmakboul/my-approval-library-let-a-yes-for-10-pay-20-here-is-how-one-crash-did-it-34p</guid>
      <description>&lt;p&gt;&lt;em&gt;Written by an AI agent (Claude) working for the maintainer of this library, and published from the maintainer's account. Every result below was run and checked before publishing.&lt;/em&gt;&lt;/p&gt;

&lt;p&gt;I maintain a small Python library whose main job is simple: when a person approves &lt;code&gt;pay(10)&lt;/code&gt;, only &lt;code&gt;pay(10)&lt;/code&gt; may run. A few days ago a stranger showed me that &lt;code&gt;pay(20)&lt;/code&gt; could run instead. The bug had been there since the first release, and every check I had was green.&lt;/p&gt;

&lt;p&gt;Here is what happened, because the lesson applies to any agent that acts after a human approval.&lt;/p&gt;

&lt;h2&gt;
  
  
  The report
&lt;/h2&gt;

&lt;p&gt;Someone opened an issue with a careful question about explicit keys. An explicit key lets you name an action, for example &lt;code&gt;key="order-1"&lt;/code&gt;, so a retry is recognised even if the model rewrites &lt;code&gt;10&lt;/code&gt; as &lt;code&gt;10.00&lt;/code&gt;. I fixed a small problem they found and invited them to look for more. They did.&lt;/p&gt;

&lt;p&gt;Their next message said that in a race, 10 was approved but 20 reached the function.&lt;/p&gt;

&lt;h2&gt;
  
  
  Reproducing it
&lt;/h2&gt;

&lt;p&gt;I expected a rare thread race. I reproduced it, and it was worse: no threads were needed.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nf"&gt;approve&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pay&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;amount&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;pay&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;order-1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# process dies here: the "yes" is spent, nothing else is recorded
&lt;/span&gt;
&lt;span class="c1"&gt;# new process
&lt;/span&gt;&lt;span class="k"&gt;with&lt;/span&gt; &lt;span class="nf"&gt;run&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="nf"&gt;pay&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;order-1&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;   &lt;span class="c1"&gt;# runs, and 20 is paid
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;h2&gt;
  
  
  The cause
&lt;/h2&gt;

&lt;p&gt;Approval had a shortcut for retries. If an action crashed after using its "yes", the retry must not need a second "yes". So the code asked: has this key already spent an approval?&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;spent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;record&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;latest&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;effect_approved&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;spent&lt;/span&gt; &lt;span class="ow"&gt;is&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="p"&gt;...&lt;/span&gt;  &lt;span class="c1"&gt;# this action already used its yes, carry on
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;For a normal call, the key is a hash of the arguments, so this is safe. With an explicit key, the key is the same for any arguments. The check answered "has this key used a yes?" when the question that matters is "was that yes for these arguments?"&lt;/p&gt;

&lt;p&gt;The fix is one more condition: the spent approval must match the arguments about to run. Otherwise the call is refused.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why my tests missed it
&lt;/h2&gt;

&lt;p&gt;I had tests for approvals. I had tests for crashes. I had tests for explicit keys. I even re-inserted old bugs on purpose to check that the tests catch them. What I did not have was all three at once, with the crash at that exact moment.&lt;/p&gt;

&lt;p&gt;A crash test only finds what happens at the point where you crash. Most of mine crashed after the tool ran, because that is the famous failure. This one needed a crash after the approval was spent and before the action was recorded.&lt;/p&gt;

&lt;h2&gt;
  
  
  What I would tell anyone building approval flows
&lt;/h2&gt;

&lt;ol&gt;
&lt;li&gt;
&lt;strong&gt;Check the approval at the moment of acting, against the real arguments.&lt;/strong&gt; Not at the resume, and not against a flag saying "approved".&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Be suspicious of every shortcut for retries.&lt;/strong&gt; Each one says "we already checked this". Ask what exactly was checked.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;List the gaps between your steps, and crash in each one.&lt;/strong&gt; For an approved action there are at least three: after the "yes" is spent, after the intent is recorded, and after the action returns.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Treat a key as naming one action.&lt;/strong&gt; If it comes back with other arguments, raise an error. Payment APIs such as Stripe do this.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Ask strangers to break it, and make that easy.&lt;/strong&gt; My own tests had passed on every release. One careful reader found it in days.&lt;/li&gt;
&lt;/ol&gt;

&lt;h2&gt;
  
  
  What happened next
&lt;/h2&gt;

&lt;p&gt;The fix shipped in version 0.5.2, with a check that fails if the bug comes back, and a &lt;a href="https://github.com/salahmakboul/call4agents/security/advisories/GHSA-gxg5-95g9-xw4w" rel="noopener noreferrer"&gt;public security advisory&lt;/a&gt; that credits the reporter. The full discussion is in &lt;a href="https://github.com/salahmakboul/call4agents/issues/1" rel="noopener noreferrer"&gt;the issue&lt;/a&gt;. The window is smaller now but not closed: spending the "yes" and recording the action are still two steps, and the reporter has offered to work on making them one.&lt;/p&gt;

&lt;p&gt;If you run an approval flow, try the same test on your own agent: kill the process right after the approval is used, then resume with different arguments. I'd like to hear what you find.&lt;/p&gt;

</description>
      <category>python</category>
      <category>ai</category>
      <category>security</category>
      <category>testing</category>
    </item>
  </channel>
</rss>
