# Hindsight Remembered the Cases; My Code Chose the Evidence

M Pranay — Tue, 29 Sep 2026 03:16:05 +0000

Hindsight Remembered the Cases; Code Chose the Evidence

The hardest part of giving an agent memory was not storing more information. It was deciding what that memory was actually allowed to prove.

I built Echo, an accounts-payable exception investigation system that combines persistent institutional memory with deterministic evidence processing. When an invoice hits an exception, Echo compares the written policy with what happened in similar historical cases, shows the conditions behind those decisions, and gives a human the evidence needed to make the final decision.

The project is open on GitHub:

Echo — AP Exception Intelligence Agent

The important architectural decision is simple:

Hindsight stores memory. Deterministic code decides what counts as evidence. The LLM explains that evidence.

That separation changed the way I designed the entire system.

The problem: policy is not always practice

Accounts payable looks straightforward until an invoice does not fit the normal process.

An invoice may have:

an amount mismatch
a missing purchase order
a GST mismatch
a suspected duplicate
a policy threshold violation
a special condition that changes how the exception is handled

The written policy can tell an AP operator what should happen.

But organizations accumulate another layer of knowledge over time:

which exceptions were actually approved
which approver handled them
which workaround was used
under which condition the approval happened
whether a particular exception pattern repeatedly appeared
whether historical practice differed from the written route

That knowledge is usually scattered across old cases, spreadsheets, emails, and human memory.

I wanted Echo to make that institutional memory queryable.

But there was an immediate problem.

If an agent retrieves five “similar” historical cases, are all five really evidence?

Not necessarily.

A case can be semantically similar while belonging to a different policy regime, amount band, condition, or exception pattern.

So Echo does not allow retrieval alone to determine the conclusion.

What Echo does

At a high level, Echo sits between the written AP policy and the human decision-maker.

The workflow is:

The architecture intentionally gives each component a narrow responsibility.

The Hindsight integration

The application keeps Hindsight behind a backend memory layer rather than spreading memory calls throughout the UI.

For example, the case expansion path verifies that Hindsight is configured before writing historical cases:

const cfg = hindsightConfig();

if (!cfg) {
  console.error(
    "Hindsight is not configured. Set HINDSIGHT_BANK_ID and HINDSIGHT_BASE_URL in .env.local"
  );
  process.exitCode = 1;
  return;
}

const cases = JSON.parse(
  readFileSync(EXPANSION_PATH, "utf8")
) as ExpansionCase[];

The important design choice is that the frontend never becomes the memory system itself.

The runtime retrieval path follows the same separation:

User / Invoice
      ↓
Identify exception family
      ↓
Recall relevant Hindsight pattern
      ↓
Read supporting case IDs
      ↓
Fetch exact historical cases
      ↓
Apply deterministic comparability
      ↓
Compute evidence + confidence
      ↓
LLM explains the evidence
      ↓
Human decision
      ↓
Retain new resolution

This gives the system a clear boundary between memory, evidence, explanation, and decision-making.

Two kinds of memory

Echo separates case memory from pattern memory.

A case represents a concrete historical event:

case_id: C12
invoice: INV-NW-1193
issue_family: amount_mismatch
approver: Priya Nair
condition: vendor annexure attached
outcome: approved

A pattern is an aggregate observation supported by multiple cases:

pattern:
fuel-surcharge exception

support_count:
5

approved_count:
5

rejected_count:
0

condition:
fuel-surcharge annexure attached

supporting_case_ids:
C02, C04, C08, C10, C12

The distinction matters because the pattern can point back to the exact historical cases that support it.

Hindsight provides persistent memory.

Echo's deterministic layer decides how that memory becomes evidence.

The retrieval boundary

A simple memory implementation could look like this:

Query
  ↓
Retrieve similar cases
  ↓
Give everything to the LLM

I did not want Echo to work that way.

Instead, the runtime path is:

Query
  ↓
Identify exception family
  ↓
Recall relevant Hindsight pattern
  ↓
Extract supporting case IDs
  ↓
Read those exact cases
  ↓
Apply comparability rules
  ↓
Exclude the current invoice
  ↓
Compute evidence
  ↓
LLM explains the result

The distinction matters.

Hindsight can find memories that are relevant to the query. That does not automatically mean every retrieved memory is valid evidence for the current invoice.

Echo therefore evaluates historical cases against deterministic rules.

The comparison considers the issue family, relevant pattern, applicable policy regime, amount or difference information when available, the side of a policy threshold, and whether the case contains enough information to be compared.

A historical case that cannot be compared reliably is not silently treated as evidence.

That became one of the central rules of the project:

Similarity finds candidates. Deterministic logic decides evidence.

Confidence is computed, not generated

I also wanted to keep confidence outside the language model.

Echo calculates the confidence level from the actual evidence.

For example:

5 comparable historical cases

5 / 5 approved

MEDIUM — SMALL SAMPLE

The language model can explain what that evidence means, but it does not invent the approval count or confidence level.

That creates a clear provenance boundary:

COMPUTED
↓
Deterministic application logic

EVIDENCE
↓
Historical Hindsight cases

SYNTHESIZED
↓
LLM explanation

This is important because an explanation can sound convincing even when the underlying evidence is weak.

Echo therefore keeps the evidence and the explanation visibly separate.

A concrete investigation

Consider:

INV-NW-1188
Northwind Freight
Amount mismatch

With memory enabled, Echo retrieves five comparable historical cases:

C02
C04
C08
C10
C12

All five were approved.

The recurring condition is:

fuel-surcharge annexure attached

Instead of simply telling the user that five invoices were “similar,” Echo exposes the evidence chain:

Exception
   ↓
Fuel-surcharge pattern
   ↓
Annexure attached
   ↓
5 comparable historical cases
   ↓
5 approvals

At the same time, Echo shows the written policy separately.

The user can therefore see:

WHAT THE POLICY SAYS

vs.

WHAT HISTORICAL PRACTICE SHOWS

The point is not to silently replace written policy with history.

The point is to expose the gap between the written route and historical practice so a human can make the final decision with better context.

Before and after persistent memory

The same invoice provides a useful comparison.

Before: written policy only

Without historical memory, Echo can still evaluate the invoice against the written rule.

The system can identify the exception and show the applicable policy route.

But it has no institutional precedent to show.

The result is effectively:

Written policy:
CFO approval required for mismatch
above INR 50,000.

Historical precedent:
None available in memory mode.

After: persistent institutional memory

With Hindsight enabled, the same investigation can include:

5 comparable historical cases

C02
C04
C08
C10
C12

5 / 5 approved

Condition:
fuel-surcharge annexure attached

That changes the investigation from a pure policy lookup into a policy-plus-practice analysis.

The user can see both sources of information rather than receiving a single opaque AI conclusion.

Memory ON vs Memory OFF

Echo also makes the memory boundary explicit.

With memory enabled:

MEMORY ACTIVE

Historical cases:
5

Historical outcome:
5 / 5 approved

With memory disabled:

WRITTEN POLICY ONLY

The system must not use historical precedent when memory is off.

This matters because a visual toggle is not enough by itself. The actual data flow has to respect the boundary.

The same invoice therefore becomes a useful test:

MEMORY ON
Policy + Historical Evidence

MEMORY OFF
Written Policy Only

The no-history case

A memory system also needs to handle the situation where there is no useful precedent.

Echo has an explicit no-history state:

NO INSTITUTIONAL PRECEDENT

In that situation, the system does not invent:

historical cases
approval counts
conditions
confidence
past outcomes

Instead, it falls back to the written policy.

That gives the system a useful failure mode:

No sufficient historical precedent was found.

Written policy remains the available guidance.

I consider that an important property of the architecture.

An agent does not need to answer every historical question.

Sometimes the correct answer is that the available memory is insufficient.

Learning from human decisions

Memory is useful only if it can grow.

When a human resolves an exception, Echo retains the new case and updates the corresponding pattern.

The update flow is:

Human resolution
       ↓
Retain new case
       ↓
Read current pattern
       ↓
Read supporting cases
       ↓
Add new case
       ↓
Recompute aggregate
       ↓
Store updated pattern
       ↓
Read back and verify

The pattern stores structured information such as:

support count
approved count
rejected count
approvers
condition
policy route
last seen
supporting case IDs

The language model can explain those facts, but it does not become the source of those facts.

That keeps the learning loop inspectable.

Echo Replay

Echo Replay uses the same evidence model to compare historical precedent under different conditions.

For example:

Current condition:
fuel-surcharge annexure attached

Historical evidence:
5 comparable cases
5 approved

The user can then inspect an alternative condition.

If historical precedent does not exist for that condition, Echo shows:

No comparable historical precedent found.

It does not manufacture an approval or rejection merely because the condition changed.

Absence of evidence remains absence of evidence.

Reflection is deliberately separated

Hindsight also provides reflection capabilities.

Echo has an opt-in reflection path for producing a concise synthesis of retrieved evidence.

However, reflection is not treated as the source of structured truth.

It should not determine:

case identity
approval count
confidence
policy truth
historical outcome

Those remain grounded in the historical memory and deterministic application logic.

The responsibility boundary is:

Hindsight
    ↓
Persistent memory

Deterministic Echo logic
    ↓
Evidence + confidence + comparability

LLM
    ↓
Explanation

Human
    ↓
Final decision

This separation made the system much easier to reason about.

What changed as the memory corpus grew

The system started with a smaller verified set of historical cases.

I then expanded the corpus with 150 additional structured cases covering:

amount mismatches
missing POs
GST mismatches
duplicate suspects

The expansion was validated before being seeded into Hindsight.

Adding more memory did not simply make retrieval “better.”

It also introduced a new problem.

The larger the memory corpus became, the more likely it was that a new query could retrieve something that looked similar but belonged to a different policy regime or condition.

That made the evidence boundary more important.

The solution was not to give the language model more historical context.

The solution was to give it a smaller and better-defined evidence window.

An honest engineering lesson

One of the biggest lessons from building Echo was that persistent memory creates a second problem after solving the first one.

The first problem is:

How do I remember what happened?

The second problem is:

How do I know whether what I remembered actually applies here?

Those are different engineering problems.

The first needs persistent memory.

The second needs deterministic rules.

That distinction changed the architecture of Echo.

Instead of:

Retrieve everything
      ↓
Ask the LLM to decide

the system became:

Retrieve
   ↓
Validate
   ↓
Compare
   ↓
Compute
   ↓
Explain

That extra boundary is what makes the memory useful without allowing memory itself to become an uncontrolled source of truth.

What I would reuse in another agent

1. Separate retrieval from evidence selection

Semantic retrieval is useful for finding candidates.

It should not automatically determine which candidates are valid evidence.

2. Keep confidence outside the LLM

Counts, sample sizes, and evidence strength are easier to verify when they come from deterministic logic.

3. Store conditions alongside outcomes

“Approved” is less useful than:

Approved when the vendor annexure was attached.

The condition explains when historical behaviour occurred.

4. Treat no precedent as a valid result

An agent should be able to say:

No sufficient historical precedent was found.

rather than manufacturing a conclusion from unrelated memories.

5. Let human decisions become future memory

The system becomes more useful when resolved cases can become part of its future institutional memory.

The architecture I would reuse

The most reusable part of Echo is not the AP domain itself.

It is the separation of responsibilities.

This pattern can extend beyond accounts payable.

The domain can change.

The responsibility boundaries can remain the same:

Memory preserves experience. Deterministic logic establishes evidence. Models explain the evidence. Humans make the final decision.

Screenshots

The screenshots are part of the technical story, not just decoration.

Investigation workspace

Memory ON vs Memory OFF

Echo Replay

Final thoughts

I started Echo thinking that the interesting engineering problem would be giving an agent long-term memory.

The harder problem turned out to be defining what that memory was actually allowed to prove.

Hindsight gave Echo persistent institutional memory.

The rest of the architecture determines how responsibly that memory is used.

The result is not a database of answers.

It is a system that remembers what happened, shows the conditions under which it happened, compares that history with written policy, and gives a human a clearer basis for the next decision.

That is the part of agent memory I find most interesting:

Not remembering everything, but remembering enough to make the next decision better grounded.

Project

Echo — AP Exception Intelligence Agent

DEV Community: M Pranay

# Hindsight Remembered the Cases; My Code Chose the Evidence

Hindsight Remembered the Cases; Code Chose the Evidence

The problem: policy is not always practice

What Echo does

The Hindsight integration

Two kinds of memory

The retrieval boundary

Confidence is computed, not generated

A concrete investigation

Before and after persistent memory

Before: written policy only

After: persistent institutional memory

Memory ON vs Memory OFF

The no-history case

Learning from human decisions

Echo Replay

Reflection is deliberately separated

What changed as the memory corpus grew

An honest engineering lesson

What I would reuse in another agent

1. Separate retrieval from evidence selection

2. Keep confidence outside the LLM

3. Store conditions alongside outcomes

4. Treat no precedent as a valid result

5. Let human decisions become future memory

The architecture I would reuse

Screenshots

Investigation workspace

Memory ON vs Memory OFF

Echo Replay

Final thoughts

Project

Hindsight Resources