An agent that crashes is annoying.
An agent that crashes and comes back is where things get dangerous.
When we started running our agents in prod...
For further actions, you may consider blocking this person and/or reporting abuse
The reframe from "how do we resume?" to "is it safe to start again?" is the whole post, and it's correct. People reach for checkpoint-and-resume because it feels like the rigorous answer, but you can't cleanly resume a non-deterministic trajectory — the model may not take the same branch the second time, so "step 3" isn't a stable coordinate. Idempotency at the effect boundary is the only thing that actually holds.
The part I'd dig into more is how you classify a tool call as safe-to-replay versus world-touching.
send_emailis obviously an effect; aGETthat hits a rate-limited or stateful API is sneakier, and a "pure" lookup that writes an audit row isn't pure at all. Do you tag that at the tool-registration level, or infer it? We landed on making every effectful tool declare an idempotency key it computes from its args, so a replay dedupes at the tool instead of relying on the runtime to reason about safety — curious whether you considered pushing the decision down to the tools rather than the orchestrator.Thanks, you put it better than I did: “step 3 isn’t a stable coordinate” is exactly it.
Today we don’t classify at all: any recorded tool call blocks the automatic replay, reads included. Deliberately blunt, for exactly the cases you list. The cost is refusing some safe replays.
Agreed that the decision belongs in the tools, with one catch: a key hashed from the args breaks when the LLM rewrites them on replay. The key has to come from stable intent, never from generated content. That’s what we do in production where we’ve built it: Outlook webhooks are deduped on the provider’s message identity, dataset imports on a SHA-256 of the content, and outbound campaigns need human approval.
What’s missing is that rule generalized to every tool, and you’ve clearly already shipped that design. I’m opening an RFC issue on tool-level idempotency keys, and I’d love your help shaping it: a comment, a counter-proposal or a PR, whatever suits you.
The project is young, so early contributors really set the direction. Every contribution is credited by name, I’m happy to pair on a first PR, and I share contributors’ work with our network. Your comment already inspired my next post, and I’d love to credit you there too.
Repo: github.com/apowerb/apowerb
Thanks for the exchange!
Anis, the list of what gets retried automatically has a third entry in the code. stream_adk_agent in core/adk_runner.py sends the same request again after a 429, up to twice, wherever in the stream the 429 arrives. A chat turn that calls send_email, gets the result and then hits the rate limit on the next model call is run again: ADK appends the user's message a second time, and the model reads a history in which the email is already sent and the same request arrives once more. Whether a second email goes out is then the model's decision, which is the call the replay rules keep for a human.
The empty-turn retry stays safe because _stream_is_empty counts any function call as content. Noting a function call as it passes through the stream, and forwarding the 429 instead of retrying once one has passed, would hold the rate-limit retry to the rule the replay endpoint already follows: a run that has called a tool is not started again until someone decides it should be.
You’re right, and thank you for reading the code this closely. I checked: stream_adk_agent re-sends the same request on a 429, up to twice, with no check on whether a function call has already gone through the stream. So the list in the post is missing an entry, and that entry breaks the rule the post is built on: after send_email, the second attempt leaves the decision to the model. I’ll add an update to the post crediting you.
Some context on why that retry exists: it was added after forwarding the 429 led the model to claim it had written a row when no tool had actually run. Your fix keeps that case intact (no function call seen yet, so retrying is safe, exactly like the empty-turn check) and only changes the case that matters: once a call has passed, forward the 429 and let a human decide.
You’ve essentially written the patch already. Would you like to open the PR? Note the function call as it passes through the stream, forward the 429 once one has been seen, plus a test that streams a functionCall then a 429 and asserts there’s no retry. I’ll review it quickly, happy to pair on it, and you’d be credited by name in the repo and in the post. If you’d rather not, no problem at all: I’ll open the issue and credit you for the find.
Repo: github.com/apowerb/apowerb
thanks for the exchange
An issue fits this better than a pull request, and one thing is worth putting in it. Forwarding the 429 hands the decision to a person only if the person can tell that a tool already ran. The paths that forward a 429 today point the other way: the test for a provider delay above the cap describes it as surfacing the original error and letting the user retry manually, and the message after the retries run out says to try again in a minute. A person who resends the request after send_email went through rebuilds by hand the history the automatic retry would have produced, with the request twice and the email already sent.
So the forwarded error could name the calls that completed in that turn, send_email here, the way the replay refusal lists what a run already did, and the new test could assert that the name is in it as well as that nothing was retried. A second test with the 429 before any function call, still retried, keeps the 7 May case from coming back if someone later simplifies the rule to forward every 429.
Moving the memory from RAM to Postgres is the right call, but it quietly reclassifies the agent’s memory as user data — and that changes who has to protect it. Once runs, tool-call records and conversation state live in Postgres, they need the same isolation you’d demand of any tenant table: if another run (or another tenant’s agent) can read a row it shouldn’t, that’s the identical failure class as a missing RLS policy — just on the tables nobody thinks to put RLS on. Two things I’d look at given this design: (1) the “every tool it calls is recorded” table is now the thing you trust to decide whether a replay is safe, so it has to be append-only and not writable on the agent/client path — otherwise a compromised run edits its own history and the safety check waves through a replay that already sent the email twice; (2) if that memory is read back through the Data API, make sure it isn’t caught by a permissive policy “because it’s internal.” The amnesia/déjà vu framing is the clearest way I’ve seen the replay problem stated — it’s the same shape as webhook idempotency, you’re answering is it safe to apply this again, not where do we resume.
Thanks, this is a sharp read. You’re right that persisting memory turns it into user data, with the same isolation bar as any tenant table.
We actually hit that exact failure class. Our ADK-native session routes originally checked that a token was valid, not which user it acted for, so the user_id in the path or body was trusted. We now enforce that the token’s subject matches the user_id on every route that reads sessions or memory, including the live WebSocket, and run records are filtered by owner on every read and replay. There’s no auto-generated Data API in front of Postgres either: tables are only reachable through the backend, so there’s no permissive policy to forget. Isolation lives in the application layer today, and RLS as defense in depth is a fair next step.
On (1): the tool record is never accepted from a client or written by the agent. It’s derived server-side from the run’s events, and if that history can’t be read, or has been deleted, the result is “unknown”, which refuses the replay. What we don’t have is a database-level guarantee, like making that column write-once for the app role. Agreed it would be stronger.
If you’d like to dig into either of those, an issue or a PR would be very welcome: github.com/apowerb/apowerb
Thanks for taking it seriously enough to check, and for the detail — enforcing that the token subject matches the user_id on every session and memory route, including the live WebSocket, is the right fix, and having no auto-generated Data API in front of Postgres removes the whole permissive-policy class. The gap your own note names is the real one: "derived from the run's events, else unknown" is a solid app-level guarantee but not a database one, and a write-once column (an INSERT-only grant for the app role, or a trigger that forbids updates) is what makes it checkable by the database instead of by convention. Happy to open an issue on apowerb with the shape if that's useful to you.
Yes please, open it, and thanks. Two details from the code so it starts on the right shape:
An INSERT-only grant won’t fit as is. The row is inserted when the run starts, with tools_executed NULL, and the column is written once at the end (settle_run). So the rule is closer to a trigger that allows NULL to a value exactly once and rejects any later change. NULL already means unknown and blocks the replay, so “set once” is the only legal transition.
The bigger part: today the app role creates agent_runs itself at startup (create_all plus ensure_columns), so it owns the table and could drop that trigger. For it to be checkable by the database and not by convention, ownership has to move to a migration role, with the app role keeping only what it needs.
If you shape the issue around those two, it’s a clean one to pick up, and it’s yours to take if you want to go as far as the PR. Credited either way.
it was a great discussion ;)
Cenk, your idea is now two issues on the repo, split the way we discussed:
the write-once trigger on tools_executed: github.com/apowerb/apowerb/issues/478
moving table ownership to a migration role: github.com/apowerb/apowerb/issues/479
Both credit you. The first one is small, self-contained and fully testable, and it’s the guarantee that decides whether an agent can safely replay a failed run in production. Few first PRs touch something that central. If you want it, it’s yours: the tech team will review it and help you along the way.
Thanks again for pushing on this, your comment made the design better.
Eight restarts a day feels low until you realize one of them loses a half-finished task. The Python dict default in ADK is an easy trap.
Exactly, and it’s worse than one dict. ADK has three in-memory defaults: sessions, long-term memory (InMemoryMemoryService, per worker and never fed) and artifacts. We had to move all three: sessions to Postgres, memory to the app database scoped by agent and user, artifacts to S3 with a local file fallback (main.py).
You’re also right that Postgres alone doesn’t save the half-finished task. It saves the conversation. The run that was in flight still dies with the pod. What changed is what happens next: its input is stored, it can’t stay “running” forever, and an agent run that already called a tool can’t be replayed unless a human forces it.
If you’ve hit this kind of trap in your own stack, issues and PRs are very welcome, and every contributor is credited by name: github.com/apowerb/apowerb
Update: the one gap still open, resuming an interrupted workflow from the node it was on, is now an issue, credited to you: github.com/apowerb/apowerb/issues/480. Contributions welcome!