Have you noticed this before?
- The model forgets something it clearly knew earlier
- The same input gives a different answer later
- Adding a bit of “context” suddenly fixes the behavior
These aren’t model quirks.
They’re signals that state is unclear in the system design.
This post builds a clean mental model for how state, memory, and context work in AI-enabled systems—and how to design without accidental coupling.
Traditional Systems: Where State Is Clear
State in traditional applications is usually:
Stored in databases
Permanent or semi-permanent information that the system relies on.
Example: Customer records, order history, or inventory levels in an Oracle or PostgreSQL database.Cached intentionally
Temporary copies of data to improve performance or reduce repeated computations.
Example: Session data stored in Redis or frequently accessed product catalog data cached in memory.Passed explicitly through APIs
Data sent between services or components as part of a request.
Example: A service call that includes a user ID and account type to fetch specific account information.Auditable and recoverable
Every state change can be tracked and traced for debugging or compliance.
Example: Versioned financial transactions, order status changes, or audit logs that can reconstruct system behavior at any point.
Most importantly, this state is:
- Explicit – you know exactly where it lives
- Addressable – you can query, update, or delete it
- Deterministic – the same inputs produce predictable results
When something breaks, you can trace what changed and where, which makes debugging reliable.
AI challenges this—not by removing state, but by hiding where developers expect it to live.
What the Model Does Not Have
Let’s start with a hard boundary.
Models do not have memory.
A model:
- Does not remember previous requests
- Does not retain conversation history
- Does not accumulate state over time
Each call is independent.
If something feels “remembered,” your system supplied it again.
Three Concepts That Must Stay Separate
Many AI bugs come from mixing these up.
Context
Context is input.
Examples:
- Prompt text
- Instructions
- Examples
- Retrieved documents
- Conversation history you resend
Properties:
- Exists only for one request
- Token-limited
- Consumed, not stored
Think of context like function arguments, not variables.
Memory
Memory is external state you manage.
Examples:
- Stored chat history
- User preferences
- Retrieved embeddings
- Cached tool outputs
Properties:
- Lives outside the model
- Must be fetched intentionally
- Must be injected back into context
If memory feels implicit, the design is already fragile.
State
State is system-level truth.
Examples:
- Workflow progress
- Decisions made
- User-visible outcomes
- Audit logs
State must:
- Persist beyond a single request
- Be inspectable
- Be owned by the application
AI can influence state, but the actual state must be stored and controlled outside the model.
The Most Common Design Failure
Many systems accidentally treat the model as stateful.
This shows up as:
- Relying on conversation flow instead of stored data
- Assuming consistency without re-supplying context
- Letting decisions live only in generated text
The result:
- Non-reproducible behavior
- Debugging without ground truth
- Silent behavior drift
If restarting your service changes outcomes, state is leaking.
A Cleaner Mental Model
A more stable framing:
- Models compute
- Systems remember
- Applications decide
The model is a stateless function.
Your system decides:
- What context to assemble
- What outputs to persist
- What becomes authoritative state
Once this boundary is clear, complexity drops sharply.
A Simple Rule
If something matters tomorrow, it cannot live only in today’s prompt.
Store it.
Version it.
Own it.
AI can assist—but it should never quietly carry state for you.
Where This Leads Next
Context and Data Flow: Feeding AI the Right Information
That’s where the next post goes.
Top comments (0)