Anis Meziani

Posted on Oct 13

Amnesia and déjà vu: how our AI agents failed after every restart (and how we fixed it)

#ai #agents #python #postgres

An agent that crashes is annoying.

An agent that crashes and comes back is where things get dangerous.

When we started running our agents in production, we discovered that a restart could go wrong in two completely opposite ways. Sometimes the agent came back with amnesia: it had forgotten everything, the conversation, the context, what it was in the middle of doing. And sometimes, when we tried to fix that by simply relaunching the work, it came back with déjà vu: it happily redid everything from the start, including the things it had already done in the real world.

Like sending an email. Twice.

This post is about both failures, why the obvious fixes don't work, and what we ended up building. It's the part of an agent nobody puts in a demo, and it's the part that decides whether you can trust it.

Everything below comes from ApowerB, the open-source agent runtime we build at thaink², and every fix links to the real code. If you're new to the project, this overview of the architecture is a good place to start.

TL;DR

Amnesia: by default, our agent framework kept every conversation in RAM. Every restart erased everything. We moved the agent's memory into Postgres.
Déjà vu: "just relaunch it" is a trap. An agent acts on the world through tools. Relaunching blindly repeats those actions.
You can't cleanly "resume from step 3" with an LLM agent. So we stopped asking how do we resume? and started asking is it safe to start again?
Every run is now written down when it starts, with its input. Every tool it calls is recorded. A failed run can be replayed without a second thought only if we know it touched nothing. Otherwise the replay is refused, loudly, with the list of what it already did, until a human decides.

Part 1: Amnesia

Eight restarts a day, zero memory

At some point we ran an internal audit on one of our deployments, and one number jumped out: around eight restarts a day.

That's not unusual. Every deploy restarts the app. Kubernetes moves pods around. A node gets drained. In a healthy cluster, processes come and go all the time.

What was a problem: after each of those restarts, every conversation was gone. No history, no context. A user could be halfway through a task with an agent, and the next message would land on an agent that had never heard of them.

The cause was a default. We build on Google's Agent Development Kit (ADK). Each agent conversation lives in a session: the list of events that happened (what the user said, what the model answered, which tools it called, what came back). That session is the agent's short-term memory. And if you don't configure where sessions live, ADK falls back to an in-memory session service. A Python dict, basically. It works perfectly on your laptop, and it evaporates every time the process restarts.

The tab that killed the workflow

The second kind of amnesia was even sillier.

Our visual workflow builder (you chain agents on a canvas, and each one passes its output to the next) originally ran in the browser. The browser was the orchestrator. Which meant that if you closed the tab in the middle of a long workflow, the workflow just... stopped. Halfway. With no record of where it was.

Runs that left nothing behind

And the third one was the most fundamental.

Only one path in the whole product could retry a failed run: incoming emails, because the webhook queue kept the payload and an attempt counter. Everything else (a workflow launched from the canvas, a scheduled agent, an API call) left nothing behind when it failed. The workflow router literally kept its state in a Python dict that was emptied at the end of the stream.

So when something failed, we couldn't relaunch it, because we didn't know what it had been asked to do. We had traces, logs, token counts. We could see the failure in detail. We just couldn't redo the work.

That was the moment it clicked for us:

What was missing wasn't observability. It was the input.

Fix 1: give the agent a memory that survives restarts

The first fix is one setting, and it's the most important line in this post. We point ADK's session service at Postgres instead of RAM (see it in main.py):

app = get_fast_api_app(
    agents_dir=agents_dir,
    session_service_uri="postgresql+asyncpg://...",   # sessions in Postgres
    memory_service_uri=MEMORY_SERVICE_URI,             # long-term memory too
    session_db_kwargs={
        "pool_size": 3, "max_overflow": 2,              # 5 connections per worker
        "pool_recycle": 300, "pool_pre_ping": True,
    },
)

Now every session and every event is a row in the database. A pod can restart ten times a day; the conversation is still there when it comes back. And as a bonus, every conversation is now auditable, which matters a lot once real businesses run on it.

We did the same for long-term memory (what an agent remembers across conversations). The default there was also in RAM, per worker, and never fed. It now lives in the application database, scoped by agent and by user, so one user's memories never leak into another's.

Two small details that only show up in production:

Mind the connection pool. Out of the box, the session engine would open up to 15 connections per worker (SQLAlchemy's default 5 plus 10 overflow). On a database shared with other services, that adds up fast. We capped it at 5.
Memory grows forever if you let it. ADK's database session service has no retention policy: the events table gets one row per agent turn, each holding the full event as JSON. Left alone, it slowly eats the database. A small background task now deletes events older than 90 days, once a day. Sessions keep their shell; they just lose old turn details.

Fix 2: take the orchestrator out of the browser

The workflow engine moved to the server, built on ADK's own Workflow primitives (chaining, fan-out, join) instead of a home-made scheduler. Every step emits step_start, step_complete or step_error, and the run's outcome is written to the database in a finally block.

To be precise about what changed: closing the tab still stops the run. That's deliberate. A client that disconnects cancels its run, and the cancellation is propagated to everything the workflow started (sub-workflows, retry waits), so nothing keeps running unsupervised in the background. What changed is that the run no longer vanishes. It's recorded as cancelled, with its input, and a cancelled run can be replayed. If it's safe, that is. Hold that thought until Part 2.

Fix 3: write every run down before it starts

This is the one that made everything else possible. Every run (a workflow, a scheduled agent, a chat turn) now gets one row, written the moment it starts. Incoming webhooks keep their own durable queue, with its own idempotency; the two will merge the day the execution engine is unified.

Here's what that row holds (run_store.py):

Table("agent_runs", metadata,
    Column("run_id", String, primary_key=True),
    Column("trigger", String),        # workflow | schedule | chat | webhook
    Column("owner_id", String),
    Column("config", String),         # the input. No input, no replay.
    Column("input_file_path", String),
    Column("status", String),         # running | success | error | cancelled
    Column("attempts", Integer),
    Column("error_message", String),
    Column("replay_of", String),      # the run this one replays
    Column("tools_executed", String), # more on this one below
)

If the run came with a file, we copy that file to disk right away. That one came from experience: the email path already kept attachments, precisely so we could replay an email that had since been deleted from the mailbox. If you only keep a pointer to the user's file, the day it disappears, your replay is impossible.

Fix 4: memory on a diet

Persisting memory solved amnesia. It also created a new problem: an agent that remembers everything can choke on it.

ADK sends the full conversation history to the model on every turn. Now imagine a tool that converts a PDF into images so the model can read it. Each page is a base64 blob worth roughly 100k to 150k tokens. After five turns, the model has been sent the same image five times. You're paying, five times, for something it already read.

So before every model call, a callback strips large base64 payloads from every message except the latest one:

def before_model_callback(*, callback_context, llm_request):
    contents = llm_request.contents or []
    for content in contents[:-1]:            # never touch the latest turn
        for part in content.parts or []:
            fr = getattr(part, "function_response", None)
            if fr and fr.response:
                _strip_payload(fr.response)  # big base64 -> "<stripped: N chars>"
    return None

The model already looked at the image when it arrived. From the next turn on, its own text about the image is the source of truth. It still sees that the tool ran and what it produced, just not the raw bytes again.

Amnesia: solved. Our agents now remember. Which brings us to the second, sneakier problem.

Part 2: Déjà vu

The tempting fix

Once runs are written down with their input, the obvious next idea is: when a run fails, just relaunch it. Same input, same agent, go again. Fully automatic. Problem solved, right?

Picture an agent whose job is to process an incoming customer request. It reads the message, drafts a reply, sends the email, then updates the CRM. Halfway through the CRM update, the pod gets evicted.

Relaunch it from the start, and it reads the message, drafts a reply, and sends the email again. The customer now has two answers. Maybe two slightly different answers, because the model doesn't write the same thing twice.

That's the core of the problem. An agent isn't a pure function. It acts on the world through its tools: it sends emails, writes to databases, calls APIs. Those are side effects. And a blind relaunch repeats every one of them.

Why you can't just "resume from step 3"

The natural reaction is: fine, then don't restart from the beginning, resume from where it stopped. Skip the steps that are already done.

With a classic workflow engine, that works, because steps are fixed and deterministic. Step 3 is always step 3.

With an LLM agent, there is no step 3. The "plan" only exists inside that particular run, in the model's reasoning at that moment. Run it again and the model may take a different path, call the tools in a different order, or decide it needs a tool it didn't use the first time. There's no clean way to replay an agent while skipping the calls it already made, because you can't replay the reasoning that led to them.

So we stopped trying to answer "how do we resume?" and asked a different question:

How do we know whether it's safe to start again?

To answer that, you need exactly one piece of information: did this run already touch the world?

Fix 5: record every tool call (the call, not the answer)

Every agent run now records the list of tools it called. We read it straight from the session events (run_main.py):

def executed_tools(events) -> list[str]:
    """Tools called in a list of ADK events, in order.

    The call (functionCall) is what counts, not the response:
    a tool that crashes midway may have acted without ever answering.
    """
    names = []
    for event in events:
        for part in (event.get("content") or {}).get("parts") or []:
            call = part.get("functionCall") or part.get("function_call")
            if call and call.get("name"):
                names.append(call["name"])
    return names

That docstring hides the most important detail in this whole post. We count the call, not the response. If the email tool sends the email and then times out before answering, there's no response in the history. But the email is gone. If you only counted tools that answered, you'd conclude "nothing happened" and send it again.

Fix 6: the memory is what makes this possible after a crash

Here's where the two halves of this post connect.

When a run fails with an exception, its response is lost. We don't get the nice list of events back. But the session is in Postgres now. It kept everything that happened before the crash. So after a failure, we go read the session itself and extract the tool calls from it.

One subtlety: a chat session contains the whole conversation, including tools called in previous turns. Those aren't side effects of this run. So we only look at what happened after the user's last message.

Without Fix 1, this is impossible. Amnesia doesn't just lose the conversation, it destroys the evidence you need to decide whether relaunching is safe.

Fix 7: "unknown" is not "none"

The tools_executed column has three possible states, and the difference between two of them is the whole safety model:

[]: we checked, and no tool was called. Safe to relaunch.
["send_email", "update_crm"]: we checked, and these were called. Relaunching repeats them.
NULL: we couldn't check (the session was unreadable, for instance). We don't know.

It's very tempting to treat "unknown" as "probably nothing". We treat it as "probably something", because the cost of being wrong isn't symmetric. Refusing a safe relaunch costs a click. Approving an unsafe one costs a duplicate email to a customer.

Fix 8: the replay rules

With all of that in place, every replay goes through a single endpoint, POST /runs/{run_id}/replay, and the rules for replaying an agent run (a scheduled agent or a chat turn) fit in a table:

The original run...	What happens when you ask to replay it
succeeded	Refused. Its side effects already happened.
is still running	Refused. Wait for it to settle.
failed, and we know no tool was called	Replayed. Nothing to repeat.
failed after calling tools	Refused, with the list of tools it called.
failed, and we can't tell	Refused. Unknown is not none.

And the refusals aren't silent. They're an HTTP 409 with a message a human can act on:

def _refuse_if_side_effects(tools_executed: str | None) -> None:
    if tools_executed is None:
        raise HTTPException(409,
            "It is unknown whether this run called any tool before failing, "
            "so a replay could repeat a side effect. "
            "Pass force=true to run it again anyway.")
    tools = json.loads(tools_executed or "[]")
    if tools:
        raise HTTPException(409,
            f"This run already called {', '.join(tools)} before failing: "
            "a replay would call them again. "
            "Pass force=true to run it again anyway.")

force=true exists, on purpose. Sometimes repeating the action is fine, or even what you want. But that's a decision for a human who has read the message, not for an automatic retry loop at 3 a.m.

Three more rules that matter:

A replay is a new run that cites the original (replay_of). We never overwrite history. If the replay fails too, you can tell it apart from the original failure.
A replay starts in a fresh session. The original session may contain a half-finished exchange, and for chat it's the user's own conversation, where a replay has no business writing.
We never replay "almost". If the input file has disappeared from disk, the replay is refused (410 Gone). Relaunching without it would produce a run that silently differs from the one you think you're reproducing.

Fix 9: runs that die without saying goodbye

One last trap. A run marks itself as finished in a finally block. But when Kubernetes kills a pod, the finally never runs. The row stays running. Forever. And a run that's "still running" can never be replayed.

So whenever we read a run, we reconcile it: if it's been running for more than six hours with no completion recorded, we mark it error with an explicit message, "no completion was recorded within 6h; its actual outcome is unknown":

conn.execute(
    run_table.update()
    .where(run_table.c.run_id == run_id,
           run_table.c.status == "running")   # only if still running
    .values(status="error", error_message=message, finished_at=now())
)

The WHERE status = 'running' makes it safe even when several replicas reconcile the same run at the same time: only the first write lands. Six hours is deliberately generous, so we never close a genuinely long run by mistake. And once reconciled, the run goes through exactly the same safety rules as any other failure. If it had already sent that email, it won't send it again.

What does get retried automatically

None of this means nothing is automatic. Things are retried silently when, and only when, it's provably harmless:

Empty model turns. Sometimes the model returns an empty turn (it happens, especially with "thinking" models). By definition an empty turn contains no tool call, so asking again can't repeat a side effect. We retry it once before falling back.
Event-triggered runs. Incoming webhooks go through a durable queue: each event is processed once, a crashed event is picked up again by another worker, and retries back off and stop after a ceiling, so nothing spins forever.

One honest footnote on the empty-turn retry: ADK appends the user's message to the session on every request, so a retry leaves that message twice in the history. It's rare and harmless today, but it's on our watch list for very long sessions.

The mental model

If you remember one thing from this post, make it these three questions. Ask them before your agent relaunches anything:

Do I remember? Is the agent's memory (sessions, events, inputs) stored somewhere that survives a restart?
Do I know what it already did? Do I record every tool call, not just every tool answer?
Is doing it again harmless? And if I can't tell, do I refuse, or do I gamble?

Amnesia is a "no" to the first question. Déjà vu is skipping the other two.

What this doesn't solve (yet)

What we have today is safe replay: a failed run is either relaunched because we know it's harmless, or refused with an explanation. What we don't have yet is true resume: picking up a run mid-flight, reusing the results of tools that already ran instead of calling them again.

Getting there probably means making tools themselves idempotent (an idempotency key on every side-effecting call, so "send this email" twice is still one email) and caching tool results per run. That's the next frontier, and it's a hard one. If you've built it cleanly, I'd genuinely love to read about it.

Is this overkill?

It's mostly one table (agent_runs), one setting (session_service_uri), one callback, and a handful of rules. No new infrastructure.

And it's what lets our agents run in production for months, across industrial use cases, on Kubernetes with our Helm chart and fully on-prem with Docker Compose, restarting several times a day without anyone noticing. Which is exactly how it should be.

Try it yourself

The whole stack, database included, comes up with three commands from the apowerb-hosting repository:

git clone https://github.com/apowerb/apowerb-hosting.git && cd apowerb-hosting
cp .env.example .env && ./scripts/generate-secrets.sh
docker compose -f docker-compose/docker-compose.yml --env-file .env up -d

The interface is on http://localhost:3000 and the API on port 8000. Add a model key in the interface and you can start breaking agents on purpose: kill the container mid-run, restart it, and watch the session come back.

Going to Kubernetes? The same repository holds the Helm chart (also on Artifact Hub) and plain Kubernetes manifests. The Docker Compose guide and the Kubernetes guide walk through both, and I wrote up the honest version of the Helm install, traps included.

Before you let your agent relaunch itself, check this

[ ] Are sessions stored somewhere other than RAM?
[ ] Is long-term memory scoped per agent and per user?
[ ] Is every run written down, with its input, before it starts?
[ ] Are input files copied, not just referenced?
[ ] Do you record tool calls, including those that never answered?
[ ] Do you treat "unknown" as "unsafe"?
[ ] Is a replay a new run that cites the original, in a fresh session?
[ ] Do runs killed mid-flight get reconciled instead of staying "running" forever?
[ ] Are old events purged, and old payloads stripped from the context?

What I'd tell myself before we shipped

Defaults are written for demos. In-memory sessions are perfect until your first restart. Check every default that stores state.
Never trust "nothing happened" unless you checked. Unknown is not none.
A refusal with a clear message beats a clever automatic retry. Your users will forgive a button that says "this already sent an email, are you sure?". They won't forgive the second email.

Come break it with us

All of this lives in ApowerB, our open-source agent runtime (Apache 2.0). The replay logic is in run_main.py, the run table in run_store.py, and the memory diet in history_compaction.py. The comments explain the scars; go read them, and tell us where we got it wrong.

Issues and PRs are welcome. If you want a first contribution in this area, open an issue or pick an existing one, and I'm happy to pair on your first PR. And if this post was useful, a star on the repo helps us a lot.

Source code: github.com/apowerb/apowerb
Self-hosting (Docker Compose, Helm chart, Kubernetes manifests): github.com/apowerb/apowerb-hosting
Helm chart on Artifact Hub: artifacthub.io/packages/helm/apowerb/apowerb-chart
Docker image: hub.docker.com/r/apowerb/apowerb
Documentation: docs.apowerb.com
Product page: thaink2.com/apowerb
See an agent run end to end (75s): youtu.be/L3KoK0gqekY

Now I'm curious: what does your agent do when it crashes halfway through a task?
Start over, resume, give up? Tell me in the comments, I read every one.

Top comments (4)

Max Quimby • Oct 13

The reframe from "how do we resume?" to "is it safe to start again?" is the whole post, and it's correct. People reach for checkpoint-and-resume because it feels like the rigorous answer, but you can't cleanly resume a non-deterministic trajectory — the model may not take the same branch the second time, so "step 3" isn't a stable coordinate. Idempotency at the effect boundary is the only thing that actually holds.

The part I'd dig into more is how you classify a tool call as safe-to-replay versus world-touching. send_email is obviously an effect; a GET that hits a rate-limited or stateful API is sneakier, and a "pure" lookup that writes an audit row isn't pure at all. Do you tag that at the tool-registration level, or infer it? We landed on making every effectful tool declare an idempotency key it computes from its args, so a replay dedupes at the tool instead of relying on the runtime to reason about safety — curious whether you considered pushing the decision down to the tools rather than the orchestrator.

Anis Meziani • Oct 13

Thanks, you put it better than I did: “step 3 isn’t a stable coordinate” is exactly it.

Today we don’t classify at all: any recorded tool call blocks the automatic replay, reads included. Deliberately blunt, for exactly the cases you list. The cost is refusing some safe replays.

Agreed that the decision belongs in the tools, with one catch: a key hashed from the args breaks when the LLM rewrites them on replay. The key has to come from stable intent, never from generated content. That’s what we do in production where we’ve built it: Outlook webhooks are deduped on the provider’s message identity, dataset imports on a SHA-256 of the content, and outbound campaigns need human approval.

What’s missing is that rule generalized to every tool, and you’ve clearly already shipped that design. I’m opening an RFC issue on tool-level idempotency keys, and I’d love your help shaping it: a comment, a counter-proposal or a PR, whatever suits you.

The project is young, so early contributors really set the direction. Every contribution is credited by name, I’m happy to pair on a first PR, and I share contributors’ work with our network. Your comment already inspired my next post, and I’d love to credit you there too.

Repo: github.com/apowerb/apowerb

Thanks for the exchange!

Edward Izgorodin • Oct 13

Anis, the list of what gets retried automatically has a third entry in the code. stream_adk_agent in core/adk_runner.py sends the same request again after a 429, up to twice, wherever in the stream the 429 arrives. A chat turn that calls send_email, gets the result and then hits the rate limit on the next model call is run again: ADK appends the user's message a second time, and the model reads a history in which the email is already sent and the same request arrives once more. Whether a second email goes out is then the model's decision, which is the call the replay rules keep for a human.

The empty-turn retry stays safe because _stream_is_empty counts any function call as content. Noting a function call as it passes through the stream, and forwarding the 429 instead of retrying once one has passed, would hold the rate-limit retry to the rule the replay endpoint already follows: a run that has called a tool is not started again until someone decides it should be.

Anis Meziani • Oct 13

You’re right, and thank you for reading the code this closely. I checked: stream_adk_agent re-sends the same request on a 429, up to twice, with no check on whether a function call has already gone through the stream. So the list in the post is missing an entry, and that entry breaks the rule the post is built on: after send_email, the second attempt leaves the decision to the model. I’ll add an update to the post crediting you.

Some context on why that retry exists: it was added after forwarding the 429 led the model to claim it had written a row when no tool had actually run. Your fix keeps that case intact (no function call seen yet, so retrying is safe, exactly like the empty-turn check) and only changes the case that matters: once a call has passed, forward the 429 and let a human decide.

You’ve essentially written the patch already. Would you like to open the PR? Note the function call as it passes through the stream, forward the 429 once one has been seen, plus a test that streams a functionCall then a 429 and asserts there’s no retry. I’ll review it quickly, happy to pair on it, and you’d be credited by name in the repo and in the post. If you’d rather not, no problem at all: I’ll open the issue and credit you for the find.

Repo: github.com/apowerb/apowerb

thanks for the exchange