My Hindsight Debugging Agent Learned to Say "I Don't Know"

Deepika Chintamreddy — Mon, 09 Nov 2026 18:15:40 +0000

Demo video: https://youtu.be/4BKfKYZqbJA

The first version of my debugging agent told me, with full confidence, that my team had hit this exact error twice before. Ticket #1124 in March. Ticket #1389 in October. Neither ticket exists. The model made both up, and it did it because I had told it to.

This is the story of how I built DevMind, a debugging assistant with team memory, and how fixing that one bug taught me more about agent memory than the rest of the project did.

What DevMind does

A developer pastes an error message and a stack trace. DevMind searches the team's past debugging history, finds fixes that look related, and asks an LLM to write a diagnosis that uses them. If a teammate already fixed something similar, the answer says who, when, and what they changed. If nobody did, it says that too.

The stack is small on purpose:

FastAPI backend
Hindsight for the memory layer, running on Hindsight Cloud
Groq with openai/gpt-oss-120b for the reasoning
A plain HTML and JavaScript page, no framework

The full code is here: DevMind on GitHub.

Developer pastes error
        |
        v
FastAPI  /api/analyze
        |
        |-- Hindsight recall()   -> closest past bug fixes
        |-- Hindsight reflect()  -> short synthesis across them
        |
        v
LLM (Groq) -> diagnosis with "From team memory" and "Suggested" sections

Why I used Hindsight instead of a plain vector store

A debugging history is not a pile of documents. It is a set of events: who fixed what, in which file, on which day, and why. I wanted the memory layer to keep that structure instead of handing me back raw text chunks.

Hindsight does this with three calls, and the whole integration fits in a few lines. The agent memory idea is simple: the agent writes what happened, and later it reads back only what is relevant.

Retaining a fix looks like this (simplified from backend/services/memory.py):

await _client.aretain(
    bank_id=MEMORY_BANK_ID,
    content=content,                      # developer, error, root cause, fix
    document_id=make_document_id(error_type, file_path),
    timestamp=reported_on,                # when the bug was actually fixed
)

Recalling and reflecting on a new error:

result = await _client.arecall(bank_id=MEMORY_BANK_ID, query=query)
reflection = await _client.areflect(bank_id=MEMORY_BANK_ID, query=query)

I seeded the bank with ten sample bugs written by me, each with a developer name, a file path, a root cause, a fix, and a date spread over about eight weeks. Hindsight turned those ten records into roughly forty stored memories. It pulled out the people, the dates, the files, and separate facts for the cause and the fix.

The bug: an agent that lied when memory was off

I wanted a clear before and after, so I added a switch. Every analysis runs twice: once with memory off, once with memory on. The page shows both answers side by side.

My system prompt said, roughly, that the assistant has a persistent memory of every bug the team ever fixed. I used the same prompt for both runs and only removed the retrieved memories for the "off" run.

The model did what I asked. It acted like it had a history, found nothing in front of it, and filled the gap. For the JWT error it wrote that there were "two exact matches in the team's history" and cited two ticket numbers with months. Both were invented.

The fix was two prompts, not one. When memory is off, the model gets a prompt that says it has no access to any team history, must not mention past tickets, dates or teammates, and must write "No team memory available" under the memory section. The prompt used with memory on got stricter too. Facts about the team's past may only come from the retrieved text, and the fix section has to separate the two sources:

From team memory: ...
Suggested (generated, not from memory): ...

That label alone made the output much easier to trust. When I read an answer now, I can see which steps a teammate actually performed and which ones the model is proposing.

What it does now

For the JWT null-pointer error in src/middleware/auth.js, the two cards behave like this.

Without memory, the answer is generic and correct: check the token, guard the decoded payload, return a 401. It says "No team memory available" and names nobody.

With memory, it says that Priya Sharma fixed the same error in the same file on 2026-09-14, that the cause was silent token expiry plus a Redis session that still held the old token, and that the fix was an explicit expiry check, clearing the Redis cache, and an auto-logout on the frontend for 401 responses. All of that came from stored memory. The code sample underneath is labelled as generated.

The payment race condition works the same way. The memory answer points to Ananya Krishnan's fix from 2026-09-25, SELECT FOR UPDATE row locking plus an asyncio.gather test, and the generic answer only says "add a lock".

The case I like best is an error nobody on the team has seen, an SMTP authentication failure. Hindsight still returns its five closest memories, which are unrelated. The model reads them, decides they do not apply, writes "No relevant team memory found", and labels everything else as generated advice.

Two more things that went wrong

Duplicates. My first seed script ran every time I ran my test script. After a few runs the bank held 65 memories instead of about 35, and the recalled snippets started repeating each other. The fix was a stable document_id built from the error type and file path, so storing the same bug again updates it instead of adding a copy, plus a reset endpoint to clear the bank. I also stopped the test script from seeding by default. Each store call also costs credits, so duplicates were a money problem as well as a quality problem.

Recall has no "nothing relevant" signal. I tried to filter out unrelated snippets using the score on each recalled item. In the runs I checked, the final score simply dropped by a fixed step per rank, 1.0, 0.975, 0.95 and so on. It is a position, not a relevance measure. A threshold that happened to cut the unrelated CORS snippet for the JWT error would have kept four unrelated ones for a new error. I removed the filter and moved the judgement into the prompt: the retrieved memories are only the closest matches and may be unrelated, and the model should use one only if it describes the same or a similar error.

Lessons

Test the agent with memory off. If your prompt claims a history, the model will invent one the moment retrieval comes back empty.
Separate remembered facts from generated advice in the output. A simple label makes answers easier to check and easier to trust.
Give writes a stable identity. Without a document ID, every re-run of an import script quietly makes your memory worse.
Do not treat recall rank as relevance. A search always returns something. Decide what "no match" means and test it with an input you know has no match.
Store real dates. Passing the real fix date as the timestamp made the answers read like a team history instead of a list of facts.

Limits

The ten bugs are synthetic, written by me. The bank belongs to one fictional team, and I have not tested DevMind against a real team's history. Recall quality on thousands of messy tickets is an open question I cannot answer from ten records. The code samples in the answers come from the model, and I have treated them as suggestions to review, not as fixes to paste.

I wrote the code with a coding agent in my editor. I also used Band desktop to run a three-agent team: a lead, an implementer and a reviewer. The reviewer checked my repository cleanup commit item by item against the task, and the implementer wrote a short README section for the live test script, which the reviewer checked against what the script really does. The implementer kept failing with a context overflow error until I changed its model, so the cleanup edits themselves were made in my editor, not by that agent.

If you want to try it, the repo is at github.com/Deepikachintamreddy/devmind-hindsight, and the Hindsight documentation covers the memory calls in more detail.

DEV Community: Deepika Chintamreddy