How We Made an LLM Actually Use Recalled Memory

Gayathri — Sun, 27 Sep 2026 18:13:32 +0000

Making Recalled Memory Actually Influence LLM Recommendations

When we integrated Hindsight into PayEcho, retrieving a customer's history was not the difficult part.

The difficult part was getting the language model to actually use that history when generating a recommendation.

The model could see the recalled information in its context and still produce almost the same generic answer it would give to a customer with no history.

That became the main engineering problem I worked on:

How do you make recalled memory act as evidence for an LLM's recommendation instead of just additional context?

The initial approach

PayEcho is a payment-recovery and credit-decision agent.

For payment recovery, the system needs to consider things such as:

previous recovery attempts
communication channels
customer responses
payment outcomes
timing of previous follow-ups

The first version of the recommendation flow was straightforward:

Get the current invoice.
Recall the customer's previous history.
Give both to the language model.
Ask it for a recommendation.

The model produced a reasonable-looking answer.

But there was a problem.

For a customer with several months of recorded behavior, the recommendation could look almost identical to the recommendation for a customer the system had never seen before:

"Send a polite email reminder, and follow up in 3–5 business days if no response."

The historical information was present in the context.

The model simply wasn't required to use it.

Having memory is not the same as using memory

This was the distinction that became important in the implementation.

A system can technically have a memory layer:

But that does not guarantee that the generated response is based on those events.

The model can treat the recalled information as background context and fall back to a generic recommendation.

So the problem wasn't initially the memory retrieval itself.

It was the connection between retrieval and reasoning.

Making memory part of the reasoning

The change was surprisingly small.

Instead of simply asking the model to make a recommendation using the available history, the recommendation needed to cite the specific previous outcome that justified the recommendation.

For example, suppose the recalled history showed that a customer:

ignored previous email reminders
responded to WhatsApp
completed payment after a follow-up three days later

The recommendation could then be:

"ABC previously ignored email reminders but responded to WhatsApp, and completed payment after a 3-day follow-up. Recommend WhatsApp outreach with a scheduled 3-day follow-up."

That is different from simply saying:

"The customer has previous payment history."

The first recommendation identifies the evidence that influenced the decision.

This made the memory load-bearing rather than decorative.

The agent loop

The recommendation flow in PayEcho is intentionally narrow.

The important part is that the model itself does not change between these interactions.

What changes is the evidence available to it.

recall() retrieves previous recovery attempts and outcomes.

The current invoice is considered alongside that history.

The model then produces a recommendation involving things such as:

communication channel
timing
tone
the specific historical event supporting the recommendation

After the business acts, the actual outcome is written back through retain().

That outcome can then become evidence for a future recommendation.

Retrieval and generation are separate

One architectural decision that helped during development was keeping retrieval and generation as separate stages.

It would have been possible to combine everything into one step, but separating them made debugging much easier.

When a recommendation looked generic, there were two different questions:

Did recall() return useful history?

Did the model receive useful history but fail to reason from it?

With separate stages, those questions could be investigated independently.

This matters because "the agent gave a bad recommendation" doesn't necessarily mean the same thing as "the memory system failed."

The failure could be in retrieval.

It could be in how the retrieved information was presented.

Or it could be in the model's reasoning.

Separating the stages made that distinction visible.

The credit-decision case is different

PayEcho also has a credit-decision use case, but I intentionally kept the model's responsibility narrower there.

For payment recovery, the agent can recommend an action.

For a credit decision, the agent surfaces relevant evidence instead of automatically approving or denying the request.

For example, if a customer has a history of late payments and requests additional credit, the system can summarize the repayment history for the person making the decision.

The final financial decision remains with the human.

The distinction is important:

Recovery:
History → Reasoning → Recommended action

Credit decision:
History → Reasoning → Evidence for human decision

The same memory and reasoning approach can support both cases, but the authority given to the model is different.

Handling empty memory

A memory-based agent also needs to handle the situation where there simply isn't any useful history.

For a new customer, recall() can return no relevant events.

In that situation, the system shouldn't manufacture personalization.

Instead, it explicitly treats the customer as having no relevant history and uses a sensible generic recommendation.

That gives the system two clearly different states:

No relevant history
↓
Generic starting recommendation

and:

Relevant history
↓
Recommendation grounded in previous outcomes

The absence of memory is therefore an explicit state rather than something the model is expected to hide.

Failure handling matters too

LLM-based systems don't always return perfect output.

Function-calling errors, malformed responses, and rate limits are possible.

The agent layer therefore includes retry handling with backoff and a fallback recommendation rather than exposing a raw generation failure to the person using the dashboard.

The goal isn't to pretend that the model never fails.

The goal is to make failure predictable and recoverable.

What changed as history accumulated?

The behavior also changed as more interactions were retained for the same customer.

With zero prior events, the system uses its generic fallback. There isn't enough evidence to make a customer-specific recommendation.

With one or two retained events, recommendations can begin referencing a specific channel that previously worked.

By the third or fourth retained interaction, the recommendation can incorporate more of the observed behavior, including timing, tone, and channel.

For example, instead of simply recommending a reminder, the recommendation can be grounded in the customer's previously observed response pattern.

The important point is not that more memory automatically makes the model better.

It is that relevant retained outcomes give the model more evidence to reason from.

What I learned

There were three lessons that stood out from building this layer.

Don't assume the model will use memory just because it can see it

Putting recalled information into the context window isn't enough.

If memory is supposed to influence the recommendation, the generation step needs to make that relationship explicit.

Requiring the recommendation to identify the relevant past outcome was a simple way to do that.

Keep recommendation and decision-making separate

An agent can summarize evidence or recommend an action without necessarily owning the final decision.

This distinction was especially important for the credit-decision part of PayEcho.

The system can surface repayment history without automatically making a financial decision.

Design the failure path early

Empty memory, malformed output, function-calling failures, and rate limits are part of building an LLM application.

A retry strategy and an honest fallback are easier to reason about when they are designed as part of the agent architecture rather than added after something breaks.

Why this pattern is broader than payment recovery

The same problem appears anywhere an agent needs to reason over historical interactions.

support systems can recall previous customer issues
sales assistants can recall previous follow-ups
incident-response agents can recall earlier incidents
operations systems can recall previous actions and outcomes

In all of these cases, there is a difference between:

The system has memory.

and:

The system's current reasoning is grounded in that memory.

That distinction is what mattered most in PayEcho.

Final takeaway

The interesting part of adding memory to an AI agent isn't just storing and retrieving previous events.

The harder problem is making those events useful evidence for the next decision.

For PayEcho, the resulting loop is:

Retain the outcome
↓
Recall relevant history
↓
Reason using that history
↓
Make a recommendation
↓
Observe the outcome
↓
Retain again

DEV Community: Gayathri

How We Made an LLM Actually Use Recalled Memory