I Added Hindsight Memory to an ML Agent. F1 Didn't Move.

Divyadharshini V — Sat, 07 Nov 2026 10:38:24 +0000

I expected the agent with memory to win. After 120 training runs on a churn-prediction task, the mean test F1 was 0.573 for the agent without memory and 0.574 for the one with it. The difference showed up somewhere else, and one of my own bugs nearly hid it.
This is what I built, how I used Hindsight, and what the numbers did and didn't show.
The problem I wanted to fix
Training a model means running dozens of experiments: different algorithms, settings and features. A few weeks later, nobody remembers what was tried, what failed and why. People repeat dead ends, and the lessons leave when a teammate does.
I called the project Ninaivu, which means "memory" in Tamil. It is an ML lab notebook agent. It remembers every experiment, including the failures, and uses that history to propose the next run. The code is public at github.com/divya240707/Ninaivu.

How it works
The loop has four steps:
Recall earlier experiments from memory.
Propose the next configuration with an LLM (openai/gpt-oss-20b on Groq).
Run it: a real scikit-learn training run on the Telco Customer Churn dataset from Kaggle. The agent picks the model (logistic regression, random forest or gradient boosting), its hyperparameters, whether to drop columns, and whether to scale.
Retain the configuration and its result, including failures, back into memory.
A run counts as overfit when train F1 exceeds test F1 by more than 0.15. I treat overfit runs and crashes as "bad runs".
Where Hindsight fits
I used the open-source agent memory system Hindsight through its Python client, with Hindsight Cloud. The Hindsight documentation lists three operations, and I used all three.
Retain runs after every experiment:
client.retain(
bank_id=bank,
content=f"Experiment: {json.dumps(config)} -> {json.dumps(result)}",
context="ML experiment log on churn classifier",
)
Recall runs before every proposal in sessions 2 and 3:
found = client.recall(
bank_id=bank,
query="Which configurations overfit, errored, or scored best, and what should be tried next?",
)
recalled = [r.text for r in found.results[:12]]
Reflect powers the "Ask the memory" page in the demo app, where you can type a question and get an answer from the stored experiments.
One thing surprised me. Hindsight does not hand your raw log lines back. It extracts facts from what you store. Recall returned sentences like "Dropping the customerID column eliminated overfitting in the logistic regression model", not the JSON I had written. That makes the results readable, but it means you should check what comes back against what you stored.
The agent memory explanation from Vectorize describes the idea in general terms. My project is one small, concrete case of it.
The experiment
I wanted a fair test of whether memory helps, so I built two agents with the same model and the same prompt:
Memory agent: recalls from Hindsight at the start of each run in sessions 2 and 3.
Baseline agent: no persistent memory.
Each trial has 3 sessions of 10 runs, and the agent's context is wiped between sessions. That models coming back to a project weeks later. Each trial used its own memory bank, so trials never shared memories. I ran 2 trials per agent, 120 runs in total.
What the numbers showed

Test F1 did not move. Mean test F1 was 0.573 for the baseline and 0.574 for the memory agent. No session differed by more than 0.001. The F1 ceiling for these model families on this dataset looks like it sits around 0.60, which leaves little room for any agent to improve.
Bad runs moved a little. In sessions 2 and 3, the memory agent made 4 bad runs out of 40. The baseline made 9 out of 40. In session 1, where neither agent has memory, the counts were 4 and 5, so there is already a gap of one run from chance alone. With 2 trials I cannot rule out chance for the later sessions either. I treat it as a pattern worth testing, not a result.
The trap I planted didn't work. The dataset has a customerID column. A model that memorizes IDs overfits, and I expected memory to teach the agent to drop it. In a manual check, dropping it removed the logistic regression overfitting gap (train F1 0.7543 and test F1 0.6047 became 0.6031 and 0.607). But the baseline agent dropped the column in all 60 of its runs, and the memory agent in 52 of 60. The language model already knew to drop an ID column. Memory had nothing to teach it there.
The bug that almost ruined the comparison
My first set of trials looked promising: by the third session the memory agent had zero bad runs. Then I added the fallback column to my results and looked closer.
When the LLM call failed twice, my code quietly substituted a random configuration:
return random_config(), True
The failures were Groq rate limits. The free tier has a daily token cap, and my runs had used almost all of it. Nearly a third of one memory trial was random configurations, not agent decisions. I discarded that whole set, kept the file for reference, and made rate limits fatal:
except Exception as exc:
if type(exc).name == "RateLimitError":
raise SystemExit(f"rate limited: {str(exc)[:300]}")
print(f"proposal failed: {type(exc).name}: {str(exc)[:200]}")
time.sleep(2)
I reran on a different model and the new data had 0 fallback runs. The "zero bad runs" result disappeared. That one change turned an exciting chart into an honest one.
Asking the memory a question

The demo app has a page where you ask the memory a question, for example "Why should we drop the customerID column?". Next to the answer it shows the stored memories it drew from.
The first version of that page was too chatty. Its answer included the right numbers (0.7543 against 0.6047, then 0.6031 against 0.607), but it also said that dropping the column had made test scores "more stable and reliable". My own data says the opposite: the test score did not change. I rewrote the prompt to demand short answers that quote stored numbers, and I kept the retrieved memories on screen so you can check the answer against its sources.
Lessons
Test whether your baseline already knows what your memory is supposed to teach. My planted trap taught me that the base model was ahead of my idea.
Make failures loud. A silent fallback turned an API limit into fake agent behavior. Log it, count it, and stop on it.
Count wasted runs as well as scores. When a metric has a ceiling, memory may show up as fewer mistakes, not a higher number.
Put sources next to generated answers. A memory system is only useful if you can see what it remembered.
Be honest about sample size. With 2 trials per agent, I can describe patterns, not claim effects.
Limitations
One dataset and 2 trials per agent.
The demo memory bank is seeded from logged runs and a few manual checks, not from a live run.
My "repeated failure" metric only counts exact repeats, so it almost never fires, and I draw no conclusions from it.
The code, data and results are all in the Ninaivu repository. If you want to try agent memory yourself, the Hindsight documentation is the best place to start.

AIAgents #AgentMemory #Hindsight #LLM

DEV Community: Divyadharshini V

I Added Hindsight Memory to an ML Agent. F1 Didn't Move.

AIAgents #AgentMemory #Hindsight #LLM