The first time my on-call agent answered a database outage, it told me not to waste time on DNS. It even cited a past incident as proof. That incident was real, but it had nothing to do with the database. I had handed the model sixty loose facts from eleven outages, and it stitched two of them together into a story that never happened.
That bug taught me more about agent memory than anything else in the project, so here is how I built the thing and what I changed.
What the system does
Foresight is an incident-response assistant for on-call engineers. You paste an alert. It recalls similar past incidents, then answers with the matching history, the likely cause, what to check first, what not to waste time on, and a confidence level. After the incident, the engineer records what actually fixed it, and that correction becomes memory for the next alert.
The stack is small: a Streamlit UI, Groq for the language model, and Hindsight, an open-source agent memory system, self-hosted in Docker. I seeded it with synthetic history: eleven incident post-mortems and three environment facts, such as “payments-api deploys on Fridays.” Hindsight turned those fourteen documents into 86 linked memories. The Hindsight documentation covers the retain, recall and reflect calls I use.
The design choice I care about most: every post-mortem is stored with a line that says “Fixes that FAILED”. “Restarting the pods only helped for 15 minutes” is the most useful sentence in an incident write-up, and the least likely to be written down. Agent memory is only worth having if it keeps that kind of detail.
Before: the same alert without memory
I used one alert throughout: payments-api latency at 11.8 seconds, Postgres saying too many clients already, starting 14 minutes after a Friday deploy.
With memory switched off, the answer was sensible and generic. Check pg_stat_activity, compare the pool size to max_connections, review the deploy. It could describe any Postgres outage on earth.
Bug one: sixty facts and an invented story
With memory on, my first version called Hindsight’s recall and pasted everything it returned into the prompt:
python
block = "\n".join(f"- {m.text}" for m in memories)
Sixty memories, no labels. The answer cited the right incident for the cause, then said to ignore DNS and pod restarts “because those were already tried and failed.” The restart fact was true for the payments incident. The DNS part came from a different outage entirely. It also took 15 seconds, because I was sending most of a free-tier token budget on every call.
The fix was to stop discarding provenance. Each recalled memory carries the document it came from, so I group by incident before building the prompt:
python
def format_memories(memories) -> str:
groups: dict = {}
for m in memories:
groups.setdefault(m.source, []).append(m.text)
return "\n\n".join(f"[{doc}]\n" + "\n".join(f"- {t}" for t in texts)
for doc, texts in groups.items())
The system prompt now says a fact under [INC-0388] belongs only to INC-0388. I also cap recall at eight memories. The same alert now returned eight memories from four incidents, a correct citation, and no DNS tangent, in 8.5 seconds.
Bug two: recall never says “nothing”
I then fed it an alert it had never seen: an iOS app crashing on launch. Hindsight returned 51 memories, because similarity search always returns the nearest neighbours, even when none are close. The agent did not invent an incident, but it borrowed a generic lesson from an unrelated one and presented it as evidence.
Recall accepts a minimum score, so I printed the reranker scores for my four demo alerts. Real matches scored 0.21 and up. Unrelated memories scored 0.03 or less. I set a floor between them:
python
if RECALL_MIN_RERANKER:
kwargs["min_scores"] = {"reranker": float(RECALL_MIN_RERANKER)}
res = c.recall(**kwargs)
With the floor at 0.1, the iOS alert recalls nothing, the memory panel says so, and the answer begins “Matching history: None found.” That sentence is worth more to an engineer at 2 a.m. than a plausible guess.
The part that learns
After an incident, a form asks for the real root cause and what did and did not work. It is stored with the incident ID as the document ID, so re-saving updates instead of duplicating:
python
memory.retain(content, context=f"Live incident feedback {inc}",
tags=["source:feedback", f"incident:{inc}"], document_id=inc)
I saved one correction for the payments alert: a batch job in the new release leaked connections, and rolling back fixed it where raising the pool size did not. Running the same alert again put that correction, INC-L001, at the top of the memory panel, next to the original incident. Nothing was retrained. The agent simply remembered what I told it.
Lessons
Pass provenance to the model. Memory without source labels invites the model to blend incidents. Grouping by source fixed my worst bug.
Retrieval needs a way to say no. Nearest-neighbour search always answers. Look at your score distributions and set a floor, then make the empty case an explicit, honest answer.
Store what failed. Negative knowledge saves the most time and is almost never documented.
Free-tier limits shape architecture. Groq’s free tier allows about 8,000 tokens per minute, and every Hindsight retain uses the LLM for fact extraction. My first seeding run hit a 429, so seeding is now throttled and resumable and takes about 15 minutes. I also had to set the Hindsight server’s Groq service tier to on_demand.
Don’t trust a threshold from four examples. More on that below.
Limitations
The incident history is synthetic. The 0.1 cutoff comes from four demo alerts and will need re-tuning on real data. Alerts are pasted in; there is no PagerDuty or Slack integration. The agent advises and a human acts. There is one shared memory bank and no authentication. An answer takes roughly 7 to 30 seconds on Groq’s free tier, depending on rate limits. Hindsight also has a reflect operation for reasoning across incidents, and the app has a toggle for it, but I have not leaned on it because of the token limit.
Where it goes next
The next feature is contradiction handling: when feedback says a past fix stopped working, flag it as stale instead of repeating it. After that, linking each memory in the panel back to its post-mortem, and taking alerts from real tooling.
The code is at github.com/iamshubham18/foresight-oncall. If you are building an agent that needs to remember, start with the Hindsight repo and spend your first hour on provenance and on teaching retrieval to say “I don’t know.”
@code.in
Top comments (0)