DEV Community: E Gayathrireddy

How We Made an LLM Actually Use Recalled Memory

E Gayathrireddy — Sun, 27 Sep 2026 18:30:10 +0000

Making Recalled Memory Actually Influence LLM Recommendations

When we integrated Hindsight into PayEcho, retrieving a customer's history was not the difficult part.

The difficult part was getting the language model to actually use that history when generating a recommendation.

The model could see the recalled information in its context and still produce almost the same generic answer it would give to a customer with no history.

That became the main engineering problem I worked on:

How do you make recalled memory act as evidence for an LLM's recommendation instead of just additional context?

The initial approach

PayEcho is a payment-recovery and credit-decision agent.

For payment recovery, the system needs to consider things such as:

previous recovery attempts
communication channels
customer responses
payment outcomes
timing of previous follow-ups

The first version of the recommendation flow was straightforward:

Get the current invoice.
Recall the customer's previous history.
Give both to the language model.
Ask it for a recommendation.

The model produced a reasonable-looking answer.

But there was a problem.

For a customer with several months of recorded behavior, the recommendation could look almost identical to the recommendation for a customer the system had never seen before:

"Send a polite email reminder, and follow up in 3–5 business days if no response."

The historical information was present in the context.
But that does not guarantee that the generated response is based on those events.

The model can treat the recalled information as background context and fall back to a generic recommendation.

So the problem wasn't initially the memory retrieval itself.

It was the connection between retrieval and reasoning.

Making memory part of the reasoning

The change was surprisingly small.

Instead of simply asking the model to make a recommendation using the available history, the recommendation needed to cite the specific previous outcome that justified the recommendation.

suppose the recalled history showed that a customer:

ignored previous email reminders
responded to WhatsApp
completed payment after a follow-up three days later

The recommendation could then be:

"ABC previously ignored email reminders but responded to WhatsApp, and completed payment after a 3-day follow-up. Recommend WhatsApp outreach with a scheduled 3-day follow-up."

That is different from simply saying:

"The customer has previous payment history."

The first recommendation identifies the evidence that influenced the decision.

This made the memory load-bearing rather than decorative.

The important part is that the model itself does not change between these interactions.

What changes is the evidence available to it.

recall() retrieves previous recovery attempts and outcomes.

The current invoice is considered alongside that history.

The model then produces a recommendation involving things such as:

communication channel
timing
tone

The model simply wasn't required to use it.

Having memory is not the same as using memory

This was the distinction that became important in the implementation.

How We Made an LLM Actually Use Recalled Memory

E Gayathrireddy — Sun, 27 Sep 2026 18:06:00 +0000

Making Recalled Memory Actually Influence LLM Recommendations

When we integrated Hindsight into PayEcho, retrieving a customer's history was not the difficult part.

The difficult part was getting the language model to actually use that history when generating a recommendation.

The model could see the recalled information in its context and still produce almost the same generic answer it would give to a customer with no history.

That became the main engineering problem I worked on:

How do you make recalled memory act as evidence for an LLM's recommendation instead of just additional context?

The initial approach

PayEcho is a payment-recovery and credit-decision agent.

For payment recovery, the system needs to consider things such as:

previous recovery attempts
communication channels
customer responses
payment outcomes
timing of previous follow-ups

The first version of the recommendation flow was straightforward:

Get the current invoice.
Recall the customer's previous history.
Give both to the language model.
Ask it for a recommendation.

The model produced a reasonable-looking answer.

But there was a problem.

For a customer with several months of recorded behavior, the recommendation could look almost identical to the recommendation for a customer the system had never seen before:

"Send a polite email reminder, and follow up in 3–5 business days if no response."

The historical information was present in the context.
But that does not guarantee that the generated response is based on those events.

The model can treat the recalled information as background context and fall back to a generic recommendation.

So the problem wasn't initially the memory retrieval itself.

It was the connection between retrieval and reasoning.

Making memory part of the reasoning

The change was surprisingly small.

Instead of simply asking the model to make a recommendation using the available history, the recommendation needed to cite the specific previous outcome that justified the recommendation.

suppose the recalled history showed that a customer:

ignored previous email reminders
responded to WhatsApp
completed payment after a follow-up three days later

The recommendation could then be:

"ABC previously ignored email reminders but responded to WhatsApp, and completed payment after a 3-day follow-up. Recommend WhatsApp outreach with a scheduled 3-day follow-up."

That is different from simply saying:

"The customer has previous payment history."

The first recommendation identifies the evidence that influenced the decision.

This made the memory load-bearing rather than decorative.

The important part is that the model itself does not change between these interactions.

What changes is the evidence available to it.

recall() retrieves previous recovery attempts and outcomes.

The current invoice is considered alongside that history.

The model then produces a recommendation involving things such as:

communication channel
timing
tone

The model simply wasn't required to use it.

Having memory is not the same as using memory

This was the distinction that became important in the implementation.

How We Made an LLM Actually Use Recalled Memory

E Gayathrireddy — Sun, 27 Sep 2026 17:06:20 +0000

Integrating Hindsight into PayEcho took an afternoon. Getting the language model to actually use what
Hindsight recalled — instead of politely acknowledging it and then giving the same generic advice
anyway — took most of the project. That gap between "the agent has access to memory" and "the
agent's answer is visibly shaped by memory" is where almost all of the real engineering work lives, and
it's the part most write-ups about agent projects skip past on their way to the demo.
The Model I Almost Shipped
My first version of the recommendation prompt just handed the model the current invoice and the
recalled history and asked for advice. It worked in the sense that it didn't crash. It failed in the sense
that the output for a customer with three months of recorded behavior looked almost identical to the
output for a customer the system had never seen:
“Send a polite email reminder, and follow up in 3–5 business days if no response.”
The recalled history was sitting right there in the context window. The model just wasn't being forced to
reason from it — it was treating memory as optional color rather than evidence it had to justify its
answer with.
Making Memory Load-Bearing, Not Decorative
The fix wasn't a bigger model or a longer prompt. It was one explicit instruction: cite the specific past
outcome that justifies this recommendation. Once the model had to point to a concrete prior event
rather than gesture vaguely at "history," the entire character of the output changed. For a customer
who'd ignored email but responded to WhatsApp and paid after a follow-up, the agent now produced:
“ABC previously ignored email reminders but responded to WhatsApp, and completed payment
after a 3-day follow-up. Recommend WhatsApp outreach with a scheduled 3-day follow-up.”
That's the difference between an LLM that has memory available and an agent that's actually built on it.

What the Agent Loop Actually Does
Structurally, the agent's job is narrow: recall a customer's relevant history, reason over it alongside the
current invoice, and produce a recommendation with a stated justification. recall() runs first and returns
the customer's past recovery attempts and outcomes. That history gets assembled into a prompt
alongside the current situation. The model generates a recommendation — channel, timing, tone, and
the specific past event it's basing that on. The business acts on it, or a person overrides it. Whatever
actually happens gets written back through retain(), which is what makes the next round of recall()
sharper than this one. The model itself never changes between calls — what changes is the evidence
it's given, and that's deliberate.

Treating the Credit-Decision Case Differently
For recovery reminders, the agent's job is to recommend an action. For credit decisions, I gave it a
narrower job on purpose: surface the evidence, don't render the verdict. When a customer with a
pattern of late payments asks for new credit, the agent's output is a summary of their repayment history
— not an approval or a denial. I could have let the model make that call; the data was there to support
it. I didn't, because risk tolerance for actual money is a judgment a person should own, and an agent
that quietly started making that call would be solving a different, riskier problem than the one I set out to
solve. The model's role stayed consistent across both use cases: reason from memory, state the
reasoning, stop short of the decision that has real financial consequences.
Building for a Model That Will Occasionally Misbehave
Function-calling errors, malformed output, and rate limits aren't edge cases with an LLM in the loop —
they're a normal Tuesday. The agent layer wraps every generation call in a retry with backoff, and falls
back to a clear, honest default recommendation rather than surfacing a raw error to whoever's
watching the dashboard. New customers get the same explicit treatment: when recall() comes back
empty, the agent doesn't pretend to have an opinion it doesn't have. It says so, and gives a sensible
generic starting point instead — which, as a side effect, makes the contrast with a returning customer's
grounded recommendation obvious the moment you put the two side by side.
What I'd Tell Someone Building the Agent Layer Next
Don't trust a model to use memory just because you handed it memory — force it to cite what it's using,
or it will default to generic advice under a thin coat of personalization. Keep the model's authority
scoped to what it should actually decide; recommending and deciding are different jobs, and blurring
them is where agent projects start to feel unsafe rather than useful. And design the failure path before
the happy path — a retry and an honest fallback cost very little to build and are the difference between
an agent that degrades gracefully and one that just breaks in front of the person relying on it.
Why This Generalizes
None of this is specific to payment recovery. Any agent reasoning over recalled history — a support
bot, a sales assistant, an incident responder — hits the same fork: either the model treats memory as
evidence it has to justify its answer with, or it treats memory as background noise it mentions in
passing. Getting that right is a prompting and architecture decision, not a bigger-model decision, and
it's the single biggest lever between an agent that feels like a stateless chatbot with extra steps and
one that actually feels like it's learned something about the person it's talking to.
A Small Design Choice That Paid Off
I split the recommendation logic into two stages instead of one — a retrieval stage that just gathers
recalled history, and a generation stage that reasons over it. It would have been faster to fold both into
a single call. Keeping them separate meant that when a recommendation came out generic, I could
check in seconds whether recall() had actually returned relevant history or whether the model had
ignored good context. That single debug question — is this a memory problem or a reasoning problem
— turned out to be the one I asked constantly, and it's much harder to answer when retrieval and
generation are fused into one opaque step.
What the Numbers Looked Like as History Grew
The quality shift wasn't binary — it tracked how much retained history existed for a given customer.
With zero prior events, the agent's fallback recommendation was, honestly, no better than a plain LLM
call with no memory at all, and that's fine; it's meant to be a safe default, not a clever guess. With one
or two retained events, recommendations started referencing a specific channel that had worked. By the third or fourth retained interaction for the same customer, the recommendations stopped looking like advice and started looking like something closer to a case note from someone who'd actually worked this account before — timing, tone, and channel all anchored to what had demonstrably worked, not to a general best practice.

The recommendation is grounded, specific, and — importantly — defensible to a human reviewing it,
because the reasoning is stated, not implied