Why memory, context, retries, and intermediate results are turning AI features into stateful software systems
You built a chatbot.
Then one day it started remembering things, waiting for approvals, retrying failed calls, and picking up work after a crash — and nobody noticed the moment it stopped being a simple chatbot and started behaving like a distributed system.
That shift matters.
A simple AI feature often looks like this:
User → Prompt → LLM → Response
It is easy to reason about. A request comes in, the model processes it, and a response goes back.
Production AI systems rarely stay that simple.
They start remembering previous interactions. They call external tools. They wait for approvals. They produce intermediate results. They retry failed operations. They resume work after a restart. They maintain information that needs to survive longer than a single request.
At that point, the interesting architectural problem is no longer just what the model can generate.
It is what the system needs to remember, where that information lives, who can change it, and what happens when something fails halfway through.
The system has state.
And once it does, you need to design it like one.
The Prompt-Response Mental Model Is Breaking
The first useful mental model for an AI feature is simple:
Input → Model → Output
For many applications, that model is still perfectly reasonable.
A user asks a question. The application sends the request to an LLM. The model returns an answer. The application displays it.
The architecture becomes different when the model starts interacting with the world.
An agent might:
call a CRM API
query a database
create a support ticket
wait for a human approval
call another service
retry a failed operation
continue a workflow after a delay
resume after a process restart
Now the system needs to know more than the current prompt.
It needs to know where it is in the workflow.
That might mean knowing that a customer request has already been classified, that a CRM lookup has completed, that an approval is still pending, or that an external operation failed after partially completing its work.
These aren’t just conversation details.
They are state.
OpenAI’s agent architecture describes agents as systems that can independently accomplish tasks through tools, while Microsoft’s agent workflow architecture similarly treats long-running workflows, state, and execution as architectural concerns. OpenAI Agents API Microsoft Agent Framework Workflows
This changes the engineering problem.
A stateless request can often be retried.
A stateful workflow may need to know whether the previous attempt already created the record, sent the email, charged the customer, or changed the database.
That is a very different problem from generating the next sentence.
Once AI systems start operating across time, state stops being an implementation detail and becomes part of the architecture.
Context Is Not State
One of the easiest mistakes in AI architecture is treating context and state as the same thing.
They are related, but they solve different problems.
Context is what the model needs to see.
State is what the system needs to remember.
A conversation history is context. It gives the model information about what has already been said.
But suppose an agent is processing a customer request and the workflow looks like this:
Customer request received
Request classified
CRM record retrieved
Approval requested
Waiting for approval
Action executed
The fact that the workflow is currently waiting for approval is not merely context.
It is state.
The distinction becomes important when the system needs to continue later.
You might be able to reconstruct some context by sending previous messages back to the model. That does not automatically tell the application whether an approval has already been requested, whether an external operation has completed, or whether the next action is authorized.
Context answers:
What does the model need to know right now?
State answers:
What does the system need to remember about the workflow?
Those questions can have different answers.
A system can rebuild context from stored information. But it still needs a reliable source of truth for workflow state.
This is why simply adding more conversation history to a prompt does not solve the state-management problem.
More context can give the model more information.
It does not give the application ownership of the workflow.
Memory Isn’t Just a Vector Database
When people hear “AI memory,” the first thing that often comes to mind is a vector database.
That makes sense for some use cases.
You might store previous conversations, documents, preferences, or other information as embeddings and retrieve relevant pieces later.
But memory in an AI system is broader than retrieval.
A production system may need to deal with several different kinds of information:
Short-term context → Session state → Workflow state → Persistent memory → Business data
These are different kinds of information, not necessarily five physical storage layers.
Short-term context can vanish after the interaction ends. Session state might survive a few turns. Workflow state might need to survive hours or days. Persistent memory holds things like user preferences, meant to carry forward.
Business data — CRM, ERP, databases, document stores — lives under its own rules for ownership, authorization, and consistency.
This distinction matters because different types of information have different lifetimes and different ownership requirements.
A user’s preferred language might be useful for months.
An approval status might matter until a workflow finishes.
A database transaction may need to remain consistent immediately.
Putting all of these things into one generic “memory” mechanism makes the architecture harder to reason about.
Microsoft’s hosted-agent architecture, for example, treats agent state as something that can persist beyond an individual interaction rather than simply treating everything as conversational context. Microsoft hosted-agent state store
The important question is not:
Where can I store the agent’s memory?
It is:
What information does the system need to remember, for how long, and who owns it?
Once you ask those questions, “memory” stops being a feature and starts looking like an architectural decision.
State Changes How You Handle Failure
In a stateless request, failure is often straightforward.
The request failed.
Try again.
With stateful AI workflows, “try again” can be dangerous.
Imagine an agent is asked to update a CRM record and send a confirmation email.
The workflow might look like:
Read CRM → Update record → Send email
What happens if the process crashes after the CRM update but before the email is sent?
A retry cannot simply repeat everything.
The CRM update may already have happened.
The system needs to know where it stopped.
This is where concepts from distributed systems become relevant to AI workflows.
Idempotency
Operations that may be retried should be designed so that repeating them does not create unintended side effects.
For example, creating the same customer record twice because an agent retried a failed request is a very different failure from generating the same piece of text twice.
Checkpointing
Long-running workflows can save meaningful intermediate state so they can resume instead of starting from the beginning.
A checkpoint might record:
which step completed
what data was produced
which tools were called
what decision was made
what remains to be done
Microsoft’s Agent Framework includes checkpointing specifically for preserving workflow state so execution can be resumed after interruptions. Microsoft Agent Framework Checkpoints
Recovery
The system needs a defined way to continue when something fails.
That might mean retrying the operation, waiting and trying later, asking for human intervention, or moving the workflow into a failed state.
Compensation
Sometimes you cannot simply retry.
If an operation partially completed, the system may need a compensating action.
For example, if one step created a record and a later step failed, recovery might require explicitly reversing or correcting the earlier action.
The important point is that the model does not solve these problems.
The surrounding application does.
The model may decide what it thinks should happen next. The application still needs to know what already happened, what is safe to repeat, and what must never be repeated.
Once AI workflows have state, failure handling becomes part of the AI architecture.
Waiting Is Also a State
One of the easiest states to overlook is waiting.
Consider an AI workflow that prepares a discount request for a sales representative.
The agent analyzes the customer, checks the account history, calculates a recommendation, and submits the request for approval.
Then nothing happens.
The manager has not approved it yet.
The workflow is waiting.
That is not an absence of state.
It is a state.
The system needs to know:
what is waiting
who needs to respond
what decision is pending
what information was already collected
what should happen after approval
what should happen if the request is rejected
how long the request can remain pending
This becomes even more important when a workflow can pause for hours or days.
You cannot keep the entire process alive in memory and assume the same process will still exist when someone eventually responds.
The workflow needs durable state.
It also needs a clear way to resume.
This is one reason human-in-the-loop workflows are closely connected to state management. Microsoft’s workflow architecture explicitly treats human interaction and checkpoints as mechanisms for pausing and resuming long-running workflows. Microsoft Human-in-the-Loop Workflows
A useful mental model is:
Running → Waiting → Resumed
The important part is that Waiting is explicit.
If it isn’t, the system has no reliable way to distinguish between:
a workflow waiting for someone
a workflow that failed
a workflow that was cancelled
a workflow that was forgotten
That distinction becomes critical as AI systems move from answering questions to completing work.
A system that can wait is already a stateful system.
State Creates New Security Boundaries
State also creates a security problem that is easy to underestimate.
In a stateless interaction, the system mainly needs to decide whether the current request is authorized.
In a stateful workflow, authorization can change as the workflow progresses.
A user might be allowed to create a request but not approve it.
An agent might be allowed to read customer data but not modify it.
A workflow might start under one identity and later resume after a human approval.
That raises a different question:
Can this workflow, at this point in its lifecycle, under this identity, access and modify this state?
The answer cannot simply be “the agent has access.”
Access needs to be tied to the workflow, the identity, the resource, and the current operation.
This becomes particularly important when state persists for a long time.
A piece of information that was safe to access during one step may not be appropriate to expose during another.
The same applies to tools.
An agent may have access to a CRM search tool but not a CRM update operation. It may be allowed to prepare an action but require human approval before executing it.
State therefore becomes part of the security boundary.
OpenAI’s agent guidance emphasizes controlling what agents can access and do through tools, while Microsoft’s agent state-store architecture also separates persisted state from the application’s broader data and authorization model. OpenAI Agents API Microsoft hosted-agent state store
The practical rule is simple:
Never assume that because an agent can see a piece of state, it should be able to change it.
Read access, write access, approval authority, and execution authority should be explicit.
Once state survives across time, security has to survive with it.
Don’t Let the Model Become Your State Store
There is a tempting shortcut in agent design:
Let the model remember what happened and decide what to do next.
It sounds natural because the model already has the conversation history.
But the model should not become the system of record for workflow state.
Consider a simple approval workflow.
A weak architecture might look like this:
User → LLM → “Approved” → Execute Action
The application is effectively trusting the model to represent a business decision.
A stronger architecture separates interpretation from authority:
User → Application → LLM interprets request → Application validates → Approval State → Business Rule → Action → Audit Record
The difference is important.
The model can interpret language.
The application should own things such as:
identity
permissions
workflow state
business rules
transactions
approvals
retries
audit records
This separation also makes the system easier to reason about.
If an approval is required, the application should know whether approval exists.
It should not have to ask the model whether someone approved something.
If an action has already been executed, the application should know that.
It should not rely on the model remembering that it happened.
OpenAI’s agent architecture similarly places tools and application-level control around the model rather than treating the model itself as the authority over external actions. OpenAI Agents API
A useful rule is:
The model can recommend a state transition. The application decides whether that transition is valid.
This is the same boundary that matters in other enterprise systems.
A recommendation is not an authorization.
A generated response is not a transaction.
And a model’s memory is not a database.
Once those boundaries are clear, the AI component becomes easier to replace, evaluate, and control without redesigning the entire workflow.
State Makes Observability Non-Negotiable
In a simple request-response system, logging the input and output may be enough to understand what happened.
In a stateful AI workflow, it usually isn’t.
When something goes wrong, you need to answer a different question:
Why did the system reach this state?
Imagine an agent that eventually sends an email to a customer.
The final action is easy to see.
But what led to it?
Which request started the workflow?
What context did the model receive?
Which tools were called?
What did those tools return?
Which intermediate decisions were made?
Which state transitions occurred?
Was a human involved?
Was anything retried?
Which version of the workflow was running?
Without that information, debugging becomes guesswork.
This is particularly difficult with long-running workflows because the final result may happen much later than the original request.
Observability therefore needs to follow the workflow, not just the model call.
A useful trace might look something like:
Request → State Created → Model Call → Tool Call → State Updated → Approval Requested → Workflow Resumed → Validation → Action → Completed
Each transition provides useful information about what the system actually did.
This also changes how you investigate failures.
Instead of asking:
“Why did the model give this answer?”
you may need to ask:
“Why was this action reached?”
Those are very different debugging questions.
The model output is only one part of the execution history.
For stateful AI systems, observability should make the workflow reconstructable enough that an engineer can understand not only what happened, but how the system got there.
Design the State Before You Design the Agent
A useful way to design an AI workflow is to start with the state rather than the agent.
Before deciding which tools the agent should have, write down the states the workflow can actually be in.
For example:
Draft → ReadyForReview → WaitingForApproval → Approved → Executing → Completed
Then define the conditions that allow each transition.
For example:
Draft → ReadyForReview: required information is present
ReadyForReview → WaitingForApproval: review has been requested
WaitingForApproval → Approved: authorized person approves
Approved → Executing: execution conditions are satisfied
Executing → Completed: the operation succeeds
You can then define what happens when something goes wrong.
Executing → Failed
Or when someone cancels the workflow:
WaitingForApproval → Cancelled
This may look like ordinary workflow design.
It is.
And that is exactly the point.
A reliable AI agent still needs deterministic boundaries around the parts of the workflow that matter.
The model can help determine what should happen next.
The application should determine whether that transition is allowed.
This approach also makes testing easier.
Instead of testing an agent as one large black box, you can test individual transitions:
Can an unapproved request reach Executing?
Can a completed workflow return to Executing?
What happens when a tool fails?
Can the same operation be safely retried?
Can a workflow resume from WaitingForApproval after a restart?
These are software engineering questions.
The AI component adds uncertainty to some decisions, but it does not remove the need for deterministic workflow rules.
Design the state machine first. Then decide where the model belongs inside it.
The More Autonomous the Agent, the More State It Needs
A simple AI interaction might look like:
Request → Model → Answer
There is very little state to manage.
An autonomous agent looks different:
Request → Plan → Tool → Result → Decision → Tool → Approval → Resume → Validation → Action → Verification → Completion
Every additional step introduces another opportunity for the system to lose information, fail, wait, retry, or make an incorrect transition.
That means autonomy and state management grow together.
Giving an agent more autonomy isn’t just handing it more tools.
It’s giving the surrounding software stronger state management and tighter control.
For example, an agent that can only search a knowledge base has relatively limited consequences if it loses its context.
An agent that can search a CRM, modify records, send emails, create orders, and trigger workflows has a very different architecture.
It needs to know:
what it has already done
what it is currently doing
what it is allowed to do next
what requires approval
what can be retried
what must not be repeated
what happened if execution stops
This is why autonomy should not be treated as a single switch.
It is a system-design decision.
The more authority an agent has, the more carefully the application needs to define state, transitions, permissions, recovery, and observability.
The goal isn’t to eliminate state.
The goal is to make state explicit, durable, and controlled.
Practical Implementation Checklist
Before building a stateful AI workflow, ask a few simple questions:
What state does the workflow actually have?
Which state must survive a restart?
Who owns and can modify each piece of state?
Which state transitions are deterministic?
Which operations must be idempotent?
What happens if a tool fails halfway through the workflow?
Can the system reconstruct why it reached its current state?
These questions are often more valuable than starting with a decision about which agent framework or model to use.
The technology will change.
Models will change.
Agent frameworks will change.
The workflow still needs to know what happened, what is happening, and what is allowed to happen next.
The Bottom Line
Input → Model → Output is still the easiest way to picture an AI system — and it’s genuinely fine for simple features.
It falls apart the moment the system starts remembering, waiting, calling tools, deciding, and continuing work across time.
At that point, the model is one component.
The rest is software architecture.
State needs ownership.
State needs persistence.
State needs authorization.
State needs observability.
State needs recovery.
And state transitions need to be designed on purpose, not discovered in production.
The question isn’t:
“How do we make the model remember?”
It’s:
“What does the system need to remember, why, for how long, and under whose authority?”
Ask that question, and AI architecture stops looking like prompt engineering — and starts looking like software engineering.
The model generates intelligence.
The system gives that intelligence a state, a boundary, and a place in the workflow.
Top comments (0)