Saying "my agent has memory" costs nothing. Anyone can paste retrieved text into a prompt and call it a day. What I wanted to know was narrower: if I take the memory away and ask the exact same question, does the agent do different work?
So I built the test into the product. This article is about that test, what it showed, and the parts that did not look as good as I hoped.
The agent under test
WhyDesk investigates why a business metric dropped. You ask "Why did revenue drop 19% in the South region in W13?" and it works through five diagnostic tools:
check_holiday_calendarcheck_competitor_promoscheck_inventory_stockoutscheck_pricing_changescheck_etl_pipeline
It runs on FastAPI, Groq tool-calling with a model fallback chain starting at qwen/qwen3-32b, a single-page frontend, and Hindsight for memory. The investigation data is seeded: I planted five known root causes, so every answer can be checked against ground truth.
After each investigation the agent saves what it learned to Hindsight: the confirmed cause, the hypotheses it ruled out, notes on unreliable data sources, and how each stakeholder wants answers formatted. The Hindsight GitHub repository has the memory engine behind this.
The test: one switch, same question
The UI has a Memory on / Memory off switch and a "Compare memory on and off" button. Compare reruns the same question twice, once with Hindsight recall and reflect, once without.
"Memory off" is strictly stateless. It makes no recall, reflect or retain calls, so it cannot leak anything into the memory bank. That matters: if the off run quietly wrote memories, the comparison would be meaningless.
What I measured
I ran the W13 South question with the Finance Head as the stakeholder. At that point the memory bank held six memories from earlier investigations.
| Measure | Memory on | Memory off |
|---|---|---|
| Checks run | 2 of 5 | 5 of 5 |
| Order | ETL, then pricing | Default: holiday, competitor, inventory, pricing, ETL |
| Skipped | Holiday, competitor, inventory | None |
| Total time | 618 ms | 600 ms |
| Final answer | -$47,900, reseller contract repricing | -$47,900, reseller contract repricing |
Two things stand out, and only one of them is flattering.
What changed: the work. With memory, the agent checked the pipeline first, because a stored note says Monday-style drops are often ETL lag, then checked pricing, found the cause, and stopped. Three checks that earlier investigations had already ruled out never ran.
What did not change: the answer. Both runs landed on the same cause and the same dollar figure, and both came back as a short Finance Head summary. Memory did not make the answer more correct here. It also did not make the run faster: 618 ms against 600 ms, since recall and reflect take time of their own. With in-memory seeded data, each check costs almost nothing, so there is nothing to save. Against a real warehouse, where each check is a slow query, skipping three of five is where I would expect the benefit. I have not measured that, so I am not claiming it.
How memory turns into tool order
The part I wanted to inspect most was how recalled memory becomes a decision. In backend/agent.py, the tool order is derived from the text of the recalled memories and the reflection, with no condition on region, week or case:
def derive_tool_order_from_memory(memories: list[dict], reflection: str) -> list[str]:
combined = " ".join(m.get("text", "") for m in memories) + " " + (reflection or "")
text = combined.lower()
if not text.strip():
return list(DEFAULT_TOOL_ORDER)
priority_scores = {tool: 0 for tool in DEFAULT_TOOL_ORDER}
if "etl" in text or "pipeline" in text or "sync" in text or "lag" in text:
priority_scores["check_etl_pipeline"] += 30
if "pricing" in text or "discount" in text or "promo code" in text or "repricing" in text:
priority_scores["check_pricing_changes"] += 20
if "stockout" in text or "inventory" in text:
priority_scores["check_inventory_stockouts"] += 15
if "ruled out" in text and "holiday" in text:
priority_scores["check_holiday_calendar"] -= 10
if "ruled out" in text and "competitor" in text:
priority_scores["check_competitor_promos"] -= 10
return sorted(
DEFAULT_TOOL_ORDER,
key=lambda t: (-priority_scores[t], DEFAULT_TOOL_ORDER.index(t)),
)
This is the honest weak spot. It is keyword matching over memory text. It works for this demo because the stored memories use predictable words, but a memory that merely mentions "inventory" in passing would boost the inventory check. With an empty bank it falls back to the default order, so the cold start is safe. The next version should let the model, or a small classifier, read the reflection and return an order, and keep this function as the fallback.
It is still memory-driven, though: remove the memories and the order reverts to the default, which is exactly what the Memory off run shows.
What is actually stored
In the memory inspector, Hindsight sorts the saved items into types on its own. The investigation verdicts show up as observation, the Finance Head's format preference as experience, and the Monday ETL-lag note as world. I did not design that split, and it is a good reason to use a purpose-built agent memory layer instead of a plain vector store.
A typical verdict reads: in 2026-W11 for the North region, the confirmed root cause was an inventory stockout, and the holiday calendar and competitor promos were ruled out.
Each run retains synchronously, so the next question already sees the previous lesson. Asynchronous retention can make an agent look forgetful when questions arrive seconds apart.
What did not work well
- Duplicates. After the comparison run, the bank held the Finance Head's format preference twice (you can see it in the image above). The agent retains it again each time it is stated. It needs a dedupe step.
- Brittle ordering, as covered above.
- Weak contrast. The memory on and off answers look nearly identical, so the comparison view shows the difference in work, not in answer quality. If I were building a better demo, I would plant a case where the default order wastes real time.
- Seeded data. Everything here runs on planted cases, so the results prove the mechanism, not real-world accuracy.
Lessons
- Test memory by removing it. An on/off switch on the same question is the cheapest evidence that memory changes behavior.
- Make the off mode fully stateless, or the comparison lies.
- Store what was ruled out. Skipping work needs that list more than it needs the answer.
- Be specific about what improved. Mine improved the work, not the answer or the speed.
- Check numbers against tool output. Every figure in the final answer must match what the tool returned, so the model cannot invent a dollar amount.
Try it
The code is here: WhyDesk on GitHub
To add memory to your own agent, start with the Hindsight documentation and the Hindsight repository.
Thanks to Code.in for the coding tools that helped me build this.
Top comments (0)