Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
Anis, the list of what gets retried automatically has a third entry in the code. stream_adk_agent in core/adk_runner.py sends the same request again after a 429, up to twice, wherever in the stream the 429 arrives. A chat turn that calls send_email, gets the result and then hits the rate limit on the next model call is run again: ADK appends the user's message a second time, and the model reads a history in which the email is already sent and the same request arrives once more. Whether a second email goes out is then the model's decision, which is the call the replay rules keep for a human.
The empty-turn retry stays safe because _stream_is_empty counts any function call as content. Noting a function call as it passes through the stream, and forwarding the 429 instead of retrying once one has passed, would hold the rate-limit retry to the rule the replay endpoint already follows: a run that has called a tool is not started again until someone decides it should be.
You’re right, and thank you for reading the code this closely. I checked: stream_adk_agent re-sends the same request on a 429, up to twice, with no check on whether a function call has already gone through the stream. So the list in the post is missing an entry, and that entry breaks the rule the post is built on: after send_email, the second attempt leaves the decision to the model. I’ll add an update to the post crediting you.
Some context on why that retry exists: it was added after forwarding the 429 led the model to claim it had written a row when no tool had actually run. Your fix keeps that case intact (no function call seen yet, so retrying is safe, exactly like the empty-turn check) and only changes the case that matters: once a call has passed, forward the 429 and let a human decide.
You’ve essentially written the patch already. Would you like to open the PR? Note the function call as it passes through the stream, forward the 429 once one has been seen, plus a test that streams a functionCall then a 429 and asserts there’s no retry. I’ll review it quickly, happy to pair on it, and you’d be credited by name in the repo and in the post. If you’d rather not, no problem at all: I’ll open the issue and credit you for the find.
Building persistent memory for AI agents at Mnemoverse.
I write about the unglamorous half: forgetting, contradiction, and the moment an agent confidently repeats a fact you retracted.
An issue fits this better than a pull request, and one thing is worth putting in it. Forwarding the 429 hands the decision to a person only if the person can tell that a tool already ran. The paths that forward a 429 today point the other way: the test for a provider delay above the cap describes it as surfacing the original error and letting the user retry manually, and the message after the retries run out says to try again in a minute. A person who resends the request after send_email went through rebuilds by hand the history the automatic retry would have produced, with the request twice and the email already sent.
So the forwarded error could name the calls that completed in that turn, send_email here, the way the replay refusal lists what a run already did, and the new test could assert that the name is in it as well as that nothing was retried. A second test with the 429 before any function call, still retried, keeps the 7 May case from coming back if someone later simplifies the rule to forward every 429.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Anis, the list of what gets retried automatically has a third entry in the code. stream_adk_agent in core/adk_runner.py sends the same request again after a 429, up to twice, wherever in the stream the 429 arrives. A chat turn that calls send_email, gets the result and then hits the rate limit on the next model call is run again: ADK appends the user's message a second time, and the model reads a history in which the email is already sent and the same request arrives once more. Whether a second email goes out is then the model's decision, which is the call the replay rules keep for a human.
The empty-turn retry stays safe because _stream_is_empty counts any function call as content. Noting a function call as it passes through the stream, and forwarding the 429 instead of retrying once one has passed, would hold the rate-limit retry to the rule the replay endpoint already follows: a run that has called a tool is not started again until someone decides it should be.
You’re right, and thank you for reading the code this closely. I checked: stream_adk_agent re-sends the same request on a 429, up to twice, with no check on whether a function call has already gone through the stream. So the list in the post is missing an entry, and that entry breaks the rule the post is built on: after send_email, the second attempt leaves the decision to the model. I’ll add an update to the post crediting you.
Some context on why that retry exists: it was added after forwarding the 429 led the model to claim it had written a row when no tool had actually run. Your fix keeps that case intact (no function call seen yet, so retrying is safe, exactly like the empty-turn check) and only changes the case that matters: once a call has passed, forward the 429 and let a human decide.
You’ve essentially written the patch already. Would you like to open the PR? Note the function call as it passes through the stream, forward the 429 once one has been seen, plus a test that streams a functionCall then a 429 and asserts there’s no retry. I’ll review it quickly, happy to pair on it, and you’d be credited by name in the repo and in the post. If you’d rather not, no problem at all: I’ll open the issue and credit you for the find.
Repo: github.com/apowerb/apowerb
thanks for the exchange
An issue fits this better than a pull request, and one thing is worth putting in it. Forwarding the 429 hands the decision to a person only if the person can tell that a tool already ran. The paths that forward a 429 today point the other way: the test for a provider delay above the cap describes it as surfacing the original error and letting the user retry manually, and the message after the retries run out says to try again in a minute. A person who resends the request after send_email went through rebuilds by hand the history the automatic retry would have produced, with the request twice and the email already sent.
So the forwarded error could name the calls that completed in that turn, send_email here, the way the replay refusal lists what a run already did, and the new test could assert that the name is in it as well as that nothing was retried. A second test with the 429 before any function call, still retried, keeps the 7 May case from coming back if someone later simplifies the rule to forward every 429.