I Put Memory Before the LLM: Building a Support Agent That Learns From Failed Fixes.

A customer should not have to explain the same problem every time they contact support. More importantly, they should not have to try the same troubleshooting step repeatedly because an AI agent forgot that it already failed.

That was the engineering problem behind RECALL AI.

I built RECALL AI around a simple idea: a support agent needs more than conversation history. It needs a reliable way to remember what happened, distinguish attempted actions from confirmed outcomes, and use that information before deciding what to recommend next.

The central design decision was to make Hindsight part of the response-generation workflow, not an afterthought attached to the conversation.

Conversations are records; memory is an interpretation

A conventional support system is good at storing messages. It can show that a customer reported a problem, that an agent suggested a fix, and that the customer replied. But storing those messages does not automatically give the next agent a useful understanding of the situation.

Consider a customer whose Bluetooth headphones disconnect every ten minutes.

During the first interaction, the agent recommends restarting Bluetooth. The customer tries it and reports that the disconnections continue. In a later session, the customer returns with the same problem.

A transcript contains all the necessary words, but a useful support agent needs to recover a more structured picture:

• The customer has a recurring Bluetooth disconnection problem.
• Restarting Bluetooth was attempted.
• The customer reported that the attempt did not solve the problem.
• The issue is still unresolved.

Those statements are related, but they are not interchangeable. In particular, an agent suggesting an action is not evidence that the customer tried it. A customer trying an action is not evidence that it worked. And silence after a recommendation is not confirmation of resolution.

I treated those distinctions as core application behavior rather than leaving them entirely to the language model.

How RECALL AI is put together

RECALL AI has four main pieces.

The RECALL AI overview dashboard (synthetic demo data). The reasoning graph at the bottom traces one customer's memory: subject fact, failed attempt, retention rule, and next step.

The React, TypeScript, and Vite frontend provides the support workspace: inbox, customer records, conversation history, memory exploration, activity, and analytics. It talks to a FastAPI backend responsible for request validation, identity and workspace checks, and the support workflow.

Supabase PostgreSQL stores operational records—workspaces, customers, conversations, messages, tickets, and agent activity. Hindsight has a different job: it retains information that may matter later and retrieves relevant memories for a new interaction. A relational database is a good system of record; it is not, by itself, a retrieval strategy for finding the few historical facts that matter now.

Groq provides language-model inference. The model receives the current request and a bounded context prepared by the application, not an unrestricted dump of every conversation.

That division of responsibility is important. Supabase answers, “What happened in the application?” Hindsight helps answer, “Which parts of what happened should influence this interaction?” The language model uses that context to formulate a response.

The most important ordering decision

The most consequential design choice was to recall memory before generating a response.

It is tempting to call the model first and then save the conversation or extract memories afterward. That can be useful for summarisation, but it does not solve the immediate problem: the model needs to know what has already failed before it recommends the next action.

The support workflow therefore follows this order:

1. Validate the request and resolve the customer's verified identity and workspace.
2. Persist the incoming conversation activity.
3. Ask Hindsight for memories relevant to the current message and customer.
4. Inspect the retrieved memories and prepare a bounded context.
5. Generate a structured response with the language model.
6. Validate the response against known failed-action constraints.
7. Persist the response and retain newly established information for future interactions.

The recall call is explicit in the agent workflow:

async with log_async_stage("hindsight_memory_recall", timing_correlation_id):
    recall_result = await self.memory_service.recall_memories(
        workspace_id,
        identity.subject_id,
        current_message,
        subject_type=subject_type,
    )

memory_context = self._memory_context(recall_result)

The important detail is not just that recall_memories exists. It is where it runs. The result is converted into application context before the generation step begins.

I keep the context bounded: retrieval is a selection step, not permission to turn the entire customer history into a prompt.

This is where Hindsight's retain, recall, and reflect model is useful. Retain gives the system a way to preserve information from an interaction. Recall retrieves relevant information for a later request. Reflect can synthesise what has been retained into a more useful understanding. I use those capabilities as memory operations, while keeping application policy—identity, provenance, validation, and response constraints—in the backend.

The Hindsight project on GitHub and the Hindsight documentation describe the memory system in more detail. The broader distinction between storing conversation text and maintaining useful agent memory is also explained in Vectorize's overview of agent memory.

Memory needs identity boundaries

Persistent memory becomes a security concern as soon as more than one customer or workspace uses the system.

I did not want raw customer identifiers to become memory-bank names. Instead, RECALL AI derives opaque, deterministic identifiers using HMAC-SHA256. The derivation includes the workspace, subject type, and canonical subject identifier:

message = "\0".join(
    ("workspace", workspace_id, "subject", subject_type, canonical_id)
).encode("utf-8")

digest = hmac.new(
    self._bank_id_secret.encode("utf-8"),
    message,
    hashlib.sha256,
).hexdigest()

return f"{self.bank_prefix}-{digest}"

This gives the application a stable memory-bank identifier for the same subject in the same workspace, while avoiding the exposure of the original customer identifier in the bank name. Including the workspace and subject type also prevents accidental reuse of the same bank identity across different namespaces.

There is an important boundary here: an opaque bank identifier is not authentication. It does not prove that a caller is allowed to access a customer's memory. The backend must still verify the request identity, workspace membership, and authorization before it asks Hindsight to read or write anything. The HMAC derivation reduces identifier exposure and helps enforce consistent scoping; it does not replace access control.

I also initialise the Hindsight client centrally rather than constructing clients throughout the application:

def create_hindsight_client(settings: Settings) -> Hindsight | None:
    if not settings.hindsight_api_key:
        return None

    return Hindsight(
        base_url=str(settings.hindsight_base_url),
        api_key=settings.hindsight_api_key.get_secret_value(),
        timeout=20.0,
        max_attempts=1,
    )

A single factory keeps endpoint, credential, and timeout configuration in one place. It also gives the application a clear integration boundary that can be tested without coupling every service to client construction.

The difficult part: remembering what an action means

The memory system is only as useful as the facts the application gives it.

For support work, I distinguish at least five kinds of information:

• A problem reported by the customer.
• A troubleshooting action suggested by the agent.
• An action the customer says they attempted.
• An outcome the customer reports, such as success or failure.
• A resolution explicitly confirmed by the customer.

This distinction prevents a subtle but damaging failure mode: treating an agent's suggestion as if it were a completed troubleshooting attempt.

For example, if the agent says, “Try forgetting the headphones and pairing them again,” the system should not immediately remember that pairing failed. It should remember that the step was recommended. Only after the customer reports trying it can the system record an attempt, and only after the customer reports the result can it classify the outcome.

The same rule applies to successful fixes. A customer saying “I'll try that” is not a resolution. A customer saying “That fixed it” is.

I use provenance checks when preparing memory context and response validation when the model returns its answer. Retrieval can surface a failed action, but the model can still overlook it or recommend it in different language. The application checks generated recommendations against known failed-action information.

I do not treat this as a problem that prompt wording alone can solve. Prompts are useful for guiding the model, but important workflow constraints belong in code where they can be tested.

A support interaction across three sessions

Here is how the workflow behaves in practice.

Session one: Sarah reports that her Bluetooth headphones disconnect roughly every ten minutes. The agent asks a few clarifying questions and suggests forgetting the device and pairing it again. The system retains the reported symptom and the fact that the action was recommended. It does not mark the action as failed because Sarah has not reported an outcome yet.

Session two: Sarah returns and says that she forgot the device, paired it again, and the headphones still disconnect. The application can now associate an attempted action with a customer-reported unsuccessful outcome. That information is retained as part of her support memory.

Session three: Sarah starts a new conversation: “My Bluetooth headphones disconnected again. Based on my previous support history, what different troubleshooting step should I try?”

The backend verifies her identity and workspace, then recalls relevant memories from Hindsight before calling the language model. The prepared context tells the agent that restarting Bluetooth and re-pairing have already been tried without resolving the issue. The agent can move to a different diagnostic direction, such as checking headphone firmware and the operating system for updates, rather than cycling through the same basic advice.

The support inbox running the Sarah scenario. Each agent reply shows the persistent memory identifiers returned by the backend, so it is visible when memory was actually retrieved.

The value is not that the agent has a longer transcript. It is that the system can use the outcome of previous actions to constrain what it recommends now.

What I learned

1. Do not confuse a transcript with a state model

Conversation history is evidence, not automatically a usable representation of the customer's situation. For workflows involving actions, outcomes, and unresolved work, model those distinctions explicitly. Otherwise, the application asks the language model to infer state from loosely related messages every time.

2. Retrieve before generation when history changes the decision

If prior interactions should affect the next recommendation, retrieve that context before asking the model to answer. Post-generation memory updates are useful, but they cannot prevent a response from repeating a known failed step in the current turn.

3. Keep memory separate from the system of record

I do not use Hindsight as a replacement for the application's relational database. Supabase owns durable application records and relationships. Hindsight provides a memory interface for retaining and retrieving useful context. Keeping those responsibilities separate makes both systems easier to reason about.

4. Preserve provenance, especially around outcomes

“Recommended,” “attempted,” “failed,” and “resolved” are different states. If the application collapses them into one generic memory, it will eventually teach the agent something that never happened. Store who reported an outcome and what evidence supports it.

5. Put critical constraints outside the prompt

A prompt can tell a model not to repeat failed actions, but application-level validation gives the system another opportunity to catch a violation. I prefer explicit checks for important workflow rules over relying on the model to follow instructions perfectly.

The part I would keep investing in

The next area I would focus on is memory-quality evaluation. A system can retrieve something related but not useful, omit an important outcome, or surface stale information. Those are separate failure modes.

I would test concrete histories: conflicting reports, recommendations never attempted, attempts with no reported outcome, and fixes that later stopped working. Each case should evaluate both what Hindsight retrieves and what the agent recommends. I would also expand the action vocabulary and response validator together, so paraphrased versions of known failed steps do not slip through.

RECALL AI started with a narrow frustration: customers repeating themselves and agents repeating fixes. The engineering lesson was broader. Persistent memory is not just about retaining more text. It is about preserving the meaning and provenance of past events, retrieving the right parts at the right time, and making sure that context changes the behavior of the system.

That is why I put Hindsight before the LLM. The agent should remember what happened before it decides what to say next.