Hindsight as a witness, not an oracle

Aditi Rajesh — Sat, 07 Nov 2026 17:01:36 +0000

Most decision support systems fail the same way: they let their AI components touch the numbers.

I built WHATIF, a business decision simulator that runs 5,000 Monte Carlo worlds to forecast the profit impact of pricing, ad spend, promotions, and restocking decisions. After grading a prediction against actual outcomes, the system updates its beliefs using closed-form Bayesian math and learns. Throughout all of this, Hindsight plays a deliberately constrained role. It is not the brain of the system. It is the system's memory — and that distinction turns out to matter a lot.

What WHATIF Actually Does

The core loop is simple: before you make a business decision, you simulate it. WHATIF runs your planned lever — raise Core prices 10%, add ₹2L in ad spend, run a 4-week promotion — through a NumPy-based simulator that samples elasticity, ad saturation curves, promo lift, and supplier delivery ratios from belief distributions calibrated to your historical decisions. You get back a p10/p50/p90 profit range, a loss probability, and a cash fan chart. You commit that prediction. You execute the decision. Two weeks later, you come back with actual numbers, and the system grades you.

Each world in that simulation is deterministic: common random numbers across baseline, plan, and test scenarios so comparisons are purely causal. The beliefs are Bayesian — normal conjugate updates for elasticity, explicit grid updates across a Hill saturation curve for ads. All of that lives in SQLite. None of it goes near an LLM.

Hindsight's job is different. It stores the narrative around decisions: what you were trying to do, why you expected it to work, what actually happened, what you said when you graded it. It surfaces relevant precedents when you're simulating a new decision. And when you ask "why were we off last quarter?", it synthesizes an explanation from retrieved experience.

What it does not do is generate, modify, or suggest numbers.

The Boundary That Makes It Work

When I initialized the Hindsight bank, I set its mission and disposition explicitly:

MISSION = (
    "Act as the decision memory for this business. "
    "Never invent numbers. "
    "Cite the memories used and say when evidence is thin."
)

await client.acreate_bank(
    bank_id=self.bank_id,
    name=self.bank_name,
    mission=MISSION,
    disposition={"skepticism": .9, "literalism": .9, "empathy": .2},
    enable_observations=True,
    enable_temporal_retrieval=True,
    enable_graph_retrieval=True,
    reflect_mission=(
        "Explain only from retrieved business experience. "
        "Do not change numeric beliefs."
    ),
)

Disposition traits shape how Hindsight's reflect operation reasons over stored memories. Skepticism at 0.9 makes it cautious with weak evidence; literalism at 0.9 keeps it close to what was actually recorded instead of extrapolating. The reflect mission has a hard prohibition: it explains from stored experience, but it cannot update beliefs. That stays entirely in the simulator's Bayesian engine.

The same constraint runs all the way through the LLM layer for prose generation. When a simulation completes, the server owns the numeric template. If the optional Groq call is available, it can fill whitelisted placeholders — [[profit_p50]], [[loss_chance]] — but it's forbidden to emit literal digits:

def substitute(template: str, values: dict[str, str]) -> str:
    names = set(re.findall(r"\[\[([a-z0-9_]+)\]\]", template))
    if not names.issubset(values):
        raise ValueError("Explanation requested an unknown numeric placeholder")
    if re.search(r"\d", re.sub(r"\[\[[a-z0-9_]+\]\]", "", template)):
        raise ValueError("Generated prose may not contain literal digits")
    return re.sub(r"\[\[([a-z0-9_]+)\]\]", lambda item: values[item.group(1)], template)

If a model slips a digit into the generated sentence, it gets rejected and the server falls back to its own template. The financial output of the system is auditable because the numbers never pass through a generative model on the way to the user.

What Hindsight Actually Receives

Every decision in the lifecycle generates a Hindsight retain call with structured tags:

await memory.aretain(
    kind="decision",
    document_id=decision_id,
    content=decision_memory_text(decision_id, decision, prediction, "committed"),
    tags=["decision", f"lever:{decision.lever}", f"group:{decision.group}", "status:committed"],
)

The content is prose: "Decision D-A856C69C committed. Change Core price by +8%. Predicted profit change −₹2,881, 80% range −₹56,995 to ₹52,380. Chance of being worse off: 52%."

When the decision is reviewed, another retain call fires with status:reviewed, appending the actual outcome and whether beliefs changed. When you later simulate a new price decision on the same product group, the recall query hits ["decision", "lever:price", "group:core"] and surfaces the top five precedents. For example, you might see that a similar price move underperformed because a Diwali pull-forward inflated the apparent baseline. That context shapes how you read the current simulation's confidence interval — not by changing the interval itself, but by surfacing the narrative the interval can't capture.

Why the Learning Gate Matters

The Bayesian update only fires under a strict attribution gate. The review function accumulates uncertainty from multiple sources:

confounder_tags = [t for t in tags if any(w in t.lower() for w in
    ("festival", "diwali", "ad issue", "suspended", "stockout",
     "weather", "competitor", "unusual", "holiday", "noise"))]

effective_uncertainty = (
    base_uncertainty
    + 0.12 * len(confounder_tags)
    + 0.04 * max(0, len(tags) - len(confounder_tags))
)

wide = (
    effective_uncertainty > .20
    or len(tags) >= 2
    or bool(confounder_tags and effective_uncertainty > .18)
)

If the band is too wide, the outcome is recorded — in SQLite and in Hindsight — but no beliefs change. The system logs the reason: "The attribution band is too wide after unusual events. The outcome is recorded, but no beliefs changed." A festival sale that coincided with your price change is real data about your business, but it's not clean enough to update your elasticity prior. Hindsight will remember the narrative; the simulator won't confuse it for a measured signal.

This separation matters for agent memory design more broadly. When you're building a system that both learns from experience and recalls experience, those two operations need different channels. The numeric beliefs are a narrow, high-precision data structure that should update slowly and defensibly. The narrative memory is wider, richer, and more tolerant of ambiguity. Trying to route both through the same LLM-powered system collapses a useful distinction.

A Concrete Interaction

Here's a run against a D2C candle brand. I asked: "What if I raise Core prices by 10%?" WHATIF ran 5,000 worlds and came back with a central estimate of −₹3,632, a likely range of −₹61,448 to +₹55,807, and a 53% chance of being worse off. Its verdict: not a clear bet yet, so start with the smaller test, which narrows the range to −₹21,719 to +₹20,773.

With Memory On, the evidence panel surfaced a precedent: an earlier +8% Core price change, predicted at −₹2,881 with a 52% chance of being worse off. At review, the actual estimated effect was ₹5,000, but the outcome was tagged with unusual events, so the gate fired: "The attribution band is too wide after unusual events. The outcome is recorded, but no beliefs changed."

That's the witness role in one screen. Memory put last time's story in front of me. The simulator refused to learn from it. Both were right.

Lessons Learned

Assign roles to systems, not tasks. The worst architecture I've seen in LLM-augmented applications is the one where the LLM is handed a task and expected to figure out the boundary between retrieval, reasoning, and calculation. WHATIF forces those into separate components: simulator computes, Hindsight witnesses, guardrails audit, Bayesian engine learns. Each component is simpler and testable on its own.

Mission text in a memory bank is load-bearing, not documentation. Setting skepticism: 0.9 and "Never invent numbers" is not flavor text. It shapes how Hindsight reasons when it reflects over recalled memories. Getting this wrong would mean synthesized explanations that feel authoritative but are disconnected from actual stored experience.

Graceful degradation is an architectural constraint, not a nice-to-have. If the Hindsight API is unreachable, the simulator keeps running. If Groq is unavailable, the server-owned verdict template still ships. The system degrades to fewer features, not broken features. This meant every integration point needed a local fallback written alongside it, which was more work but resulted in an actually shippable system.

Noisy data needs a gate, not a filter. The confounder detection in the review loop doesn't throw away festival-period data — it stores it and quarantines it from belief updates. That's different from discarding it. Hindsight accumulates the narrative around messy outcomes; the simulator just doesn't learn from them until attribution is clean enough to trust.

The system learns slowest from its hardest questions. Promotions with confounders, ad spend during stockouts, price changes over holiday periods — these are exactly the decisions where you most want calibrated beliefs, and they're the ones the gate will block most often. That's not a bug. It's the system being honest about what it knows. The alternative is confident priors built on noise.

Built with the Code.in community.