Why My Agent Kept Forgetting Things, and How Hindsight Fixed It

Bahar Fatima — Tue, 29 Sep 2026 17:23:33 +0000

A customer wrote in for the third time about slow builds on their ARM runners, and my support agent asked, politely, what kind of runners they were using.
The agent wasn't broken. It answered every message competently. It just had no idea it had ever spoken to this person, because every conversation started from a blank context window. This post is about how I fixed that with Hindsight, and about the parts that weren't as simple as "add a memory layer."
What the system does
The service is a support agent for a hosted CI product. Customers ask about failing builds, billing, runner configuration, and quotas. The agent answers in-thread, and we want it to behave like an engineer who has worked with that customer for months: remembering their stack, the workaround we suggested last week, and the thing they said they'd try.
The code is small. Three modules:
agent/memory.py wraps the Hindsight client. One memory bank per customer, retains off the request path, recalls on it.
agent/agent.py assembles the prompt and calls the LLM.
agent/main.py is a REPL I use for poking at behavior. In production the same SupportAgent sits behind our ticket webhook.
On a single turn the flow is: take the customer's message, recall relevant memories from that customer's bank, build the prompt, get a reply, then retain the exchange in the background. Hindsight runs as a separate service (Docker locally, Postgres-backed in production). The agent only ever talks to it through the Python client.
![Architecture: the support agent, its short-term window, and Hindsight as the long-term memory service]

What I tried first, and why it failed
My first version stored raw transcripts in a table and stuffed the last N messages into the prompt. It broke in the way everyone's first version breaks: the useful fact ("they run self-hosted ARM runners") was said four conversations ago, and the last N messages were all about an invoice.
My second version put transcript chunks in a vector store and retrieved by similarity. That's better, and it's the standard agent memory starting point, but two problems showed up quickly.
First, similarity search on raw chunks retrieves text, not facts. A customer's question about "slow builds" pulled in three chunks that mentioned builds and none of the one where they'd told us their runner architecture, because that sentence was in the middle of a message about a billing dispute.
Second, time. "We're on the Team plan" and "we upgraded to Enterprise last month" are both true statements in the store. Only one is true now, and cosine similarity has no opinion on which.
Both problems are about what happens at write time, which is what pushed me toward a system that does extraction on the way in rather than treating memory as an index over raw text.
The through-line: decide what a memory is when you write it
Hindsight's retain doesn't just embed what you hand it. Per the docs, it uses an LLM to extract facts, temporal data, entities, and relationships, and normalizes them into canonical entities and indexes. Recall then runs semantic, keyword, graph, and temporal retrieval in parallel, fuses the results, and reranks them. That means the design decision that mattered for me wasn't which retrieval algorithm to use. It was what I feed into retain.
I settled on retaining whole exchanges with a timestamp and a context label:

def retain_exchange(self, customer_id: str, customer_msg: str, agent_reply: str) -> Future:
    content = f"Customer: {customer_msg}\nSupport agent: {agent_reply}"
    ts = datetime.now(timezone.utc).isoformat()
    return self._pool.submit(self._retain, bank_for(customer_id), content, ts)

def _retain(self, bank_id: str, content: str, ts: str) -> None:
    try:
        self._client.retain(
            bank_id=bank_id,
            content=content,
            context="support conversation",
            timestamp=ts,
        )
    except Exception:
        log.exception("retain failed for bank %s", bank_id)

Three choices in there are worth defending.
Exchanges, not lines. I retain the customer message and the agent reply together. Half of what's worth remembering lives in the reply: "I suggested lowering the cache TTL" is a fact about the relationship, and a customer-only log would lose it. It also lets the extraction step see the agent's commitments, which is what makes "did you get a chance to look at that?" answerable.
A real timestamp on every retain. This is the fix for the Team-versus-Enterprise problem. Because each memory carries when it happened, the temporal side of recall has something to work with, and the extracted facts stay ordered instead of piling up as equally weighted assertions. I didn't build a supersession system. I passed the timestamp and let the memory layer use it.
Retain runs in a thread pool, and failures are logged, not raised. Extraction involves an LLM call, so retain isn't instantaneous, and I don't want a customer waiting on it. I also decided that a failed retain should never fail the conversation. It's a deliberate tradeoff: I would rather lose one memory and page myself from the logs than return an error to a customer who asked a simple question.
Recall on the request path
Reads are the opposite: they block, so they have to be defensive.

def recall(self, customer_id: str, query: str, max_chars: int = 3000) -> list[str]:
    try:
        resp = self._client.recall(bank_id=bank_for(customer_id), query=query)
    except Exception:
        log.exception("recall failed for customer %s", customer_id)
        return []
    kept, used = [], 0
    for text in _texts(resp):  # already relevance-ordered by Hindsight
        if used + len(text) > max_chars:
            break
        kept.append(text)
        used += len(text)
    return kept

If recall fails, the agent answers without memory rather than not answering. That's a worse answer, but it's a working one, and there's a test pinning the behavior (test_recall_failure_degrades_to_no_memory). The character budget keeps a chatty customer's history from crowding out the actual question. Results come back relevance-ordered, so truncating from the tail drops the least useful material first.
The banks are isolated per customer: bank_for("acme-42") returns customer-acme-42. Hindsight's docs describe bank isolation as strict, with no cross-bank leakage, and that's the property I wanted. In a support product, a memory that crosses customers is not a quality bug, it's an incident. I'd rather enforce that with a bank boundary than with a metadata filter I have to remember to apply on every query.
The race I didn't see coming
Here's the bug that cost me an afternoon. In my first end-to-end test, the agent remembered everything from yesterday and forgot what the customer said thirty seconds ago:

you>   our runners are self-hosted arm64
agent> Got it. Which builds are slow?
you>   the docker image builds
agent> Which architecture are your runners?

Nothing was wrong with Hindsight. Retain is asynchronous on my side and does real extraction work on theirs, so the second message's recall ran before the first exchange had been processed. I had built a memory system with a hole exactly one turn wide.
The fix wasn't to make retain synchronous. It was to stop pretending long-term memory should cover the last minute. The agent now keeps a short verbatim window of recent turns per customer and puts them in the prompt directly:

self._recent: dict[str, deque] = defaultdict(lambda: deque(maxlen=recent_turns * 2))

def build_messages(self, customer_id: str, message: str) -> list[dict]:
    memories = self.memory.recall(customer_id, message)
    system = SYSTEM_PROMPT
    if memories:
        system += "\n\n" + MEMORY_HEADER + "\n" + "\n".join(f"- {m}" for m in memories)
    return [
        {"role": "system", "content": system},
        *self._recent[customer_id],
        {"role": "user", "content": message},
    ]

Working memory is the window. Long-term memory is Hindsight. Once I split them, both got simpler. The window handles "what did they just say," recall handles "what do we know about them," and neither has to be good at the other's job. test_recent_window_carries_previous_turn_without_memory exists so nobody "cleans this up" later.
The system prompt also tells the model what to do when memory and the present disagree: trust the customer, and say you noticed the change. Recalled facts are evidence, not ground truth, and I'd rather the agent say "you mentioned Team plan before, has that changed?" than confidently bill someone for the wrong tier.
Behavior
Here's a representative session with a returning customer. The earlier conversations happened days apart; nothing from them is in the prompt except what recall returns.

you>   builds got slow again after the weekend
agent> Last time this came up you were on self-hosted arm64 runners and we
       suggested lowering the cache TTL. Did that change stick, or did the
       runner image change over the weekend?
you>   the TTL change stuck. we did move two runners to a new host though
agent> That's the likelier culprit, then. Are the new hosts also arm64, and
       do they share the same cache volume as the old ones?

Compare that to the version that asked what runners they were using. Nothing about the model changed between the two. The difference is a system prompt with a handful of recalled lines in it.
There's also a --briefing flag that calls reflect on the customer's bank with a fixed question ("What should a support engineer know about this customer before replying?"). I use it when a human takes over a thread, so they read a synthesized summary instead of scrolling a transcript:

def briefing(self, customer_id: str) -> str:
    resp = self._client.reflect(
        bank_id=bank_for(customer_id),
        query="What should a support engineer know about this customer before replying?",
    )
    return getattr(resp, "text", None) or str(resp)

Testing it without a live server
Everything above is testable without Docker or an API key running, because CustomerMemory takes an injectable client. The suite runs against a fake:
![pytest output: 6 passed, covering bank isolation, retain scoping, the recent-turns window, recall failure handling, and the character budget]

That's what let me iterate on the prompt-assembly logic and the failure-handling paths fast, without waiting on an LLM round trip for every test run. The slower, end-to-end pass — actually hitting a running Hindsight instance and watching retain and recall behave on real conversations — happens separately, against Docker.
What I'd tell someone starting this

Memory quality is decided at write time. I lost weeks tuning retrieval over raw chunks. The improvement came from handing the memory layer structured, timestamped, contextualized input and letting it extract facts. Check what your system does on retain before you look at what it does on recall.
Separate working memory from long-term memory. Asynchronous writes leave a gap of at least one turn. Cover that gap with a plain deque in the prompt. It's boring and it works.
Writes fail quietly, reads degrade gracefully. A dropped retain and a failed recall should both be survivable. Log them somewhere you'll see, and write a test for the degraded path so it stays intact.
Draw isolation boundaries with the storage model, not with filters. One bank per customer means a cross-customer leak requires a bug in the wrong place, not a forgotten WHERE clause.
Tell the model that memory can be stale. One sentence in the system prompt about trusting the present over the past prevented more bad answers than any retrieval tweak. Where this leaves me The agent now behaves like it has worked with the customer before, and the code that produces this is under 200 lines. Most of the complexity lives in a service I don't have to maintain, and the Hindsight docs cover the parts I haven't needed yet, such as mental models and per-bank configuration. I'd rather read those than reimplement fact extraction and temporal indexing myself. If your agent keeps asking questions it already knows the answer to, look at your write path first. The Hindsight repo is MIT-licensed, and the quick start gets a server running in one Docker command.

DEV Community: Bahar Fatima

Why My Agent Kept Forgetting Things, and How Hindsight Fixed It