When people talk about making AI better, they usually talk about bigger models. More parameters, longer context, a newer version. Our team tried something different while building DejaVu, an AI agent for on-call engineers. We kept the model exactly the same and only changed one thing: we gave it a memory.
The result surprised all of us.
The experiment
DejaVu helps an engineer when something breaks in production. You paste an error, and it tells you the probable root cause and how to fix it.
To test it, we used one realistic outage. A payments service starts throwing KeyError: 'amount' right after a new deploy. Then we asked the same model, GPT-OSS 120B running on Groq, the same question twice.
Round 1: no memory. The model answered with 20% confidence: "Likely a schema mismatch after deploy. Check the code and cache." That's not wrong. It's what a smart stranger would say. But it doesn't tell you what to do at 3 a.m.
Round 2: with memory. This time the agent first searched its long-term memory of past incidents, then answered with 98% confidence: a Redis cache invalidation bug after the v3.2 deploy, leaving stale v1 keys. It gave the exact command to flush those keys, told us to restart the pods, and listed 5 past incidents where this exact fix worked.
Same model. Same question. The only difference was memory.
Where the memory comes from
The memory layer is Hindsight, an open-source agent memory system by Vectorize. We store each past incident in a Hindsight "memory bank": the error, the root cause, the fix and whether it worked.
If you're new to the idea, Vectorize has a clear explainer on what agent memory is. The short version: memory is not just a pile of documents in a prompt. The agent keeps learning facts over time, and it can recall the right ones when it needs them.
Here is the call that asks Hindsight for similar past incidents:
response = client.recall(
bank_id="dejavu-incidents",
query=error_text, # the new error the engineer pasted
budget="mid",
)
for fact in response.results:
print(fact.text, fact.scores.reranker)
Each result is a fact Hindsight remembers, with a relevance score. We group the facts by incident and send only the top matches to the model. The Hindsight docs explain recall, retain and reflect in detail.
Why a bigger model wouldn't have helped
A bigger model knows more about the world. But it still knows nothing about your company. It has never seen your services, your deploys or the bug your team fixed last March.
That's the gap memory fills. In our test data, the same cache bug appeared 6 times on 3 different services, each with a different error message. One said KeyError, another said TypeError, another said "password reset tokens rejected as expired". A human would need hours to connect those. Hindsight's semantic recall connected them in seconds, because it matches on meaning, not on exact words.
No amount of model size can recall something it was never told.
The agent gets better every time
The best part is that the memory grows. After each incident, the engineer clicks Fix worked or Fix failed. That result is saved back into Hindsight as a new memory.
During one practice run of our demo, we clicked "Fix worked" once. On the next run, that brand-new incident showed up as the best match, fixed in 1 minute. The agent had learned from us in real time.
Across our 20 recorded incidents, the average time to fix dropped from 44 minutes for the first 5 to 6 minutes for the last 5. That's 87% faster, with no change to the model.
What I'm taking away
- Context beats size. For work that depends on your own history, the right memory matters more than a bigger model.
- Show the proof. Engineers trust an answer when they can see the 5 past incidents behind it.
- Let users teach it. One click of feedback turned into a new memory. That loop is what makes the agent improve.
What surprised me most
I expected memory to help a little. I didn't expect it to change the kind of answer. Without memory, the model talked in general terms: "check the code", "check the cache". With memory, it talked like a teammate who was there last time: the exact keyspace to flush, the exact service to restart, and how long it took before.
It also became more honest. We designed it so that when an error has no proven match in memory, the agent doesn't pretend. It says there is no similar incident in memory and keeps its confidence low. A model on its own rarely admits that.
That's the real promise of agent memory for me. It's not only about being right more often. It's about knowing the difference between "I've seen this" and "I'm guessing", and telling the engineer which one it is.
Try it yourself
DejaVu is open source:
- Code: https://github.com/Pratheeksh-7S/dejavu-incident-agent
- 3-minute demo video: https://youtu.be/Nu_sktguGMk
- Hindsight: https://github.com/vectorize-io/hindsight
Next time someone says "we need a bigger model", it's worth asking first: does the model actually remember anything?
Top comments (0)