Why I Made Hindsight Question Past Invoice Decisions

Kiran Hooli — Mon, 09 Nov 2026 14:42:37 +0000

An Invoice Can Be Wrong. But What Happens Next?

Imagine a finance team receiving an invoice from a supplier.

The purchase order says ₹1,800 per unit, but the supplier's invoice says ₹1,872. Detecting that 4% difference is straightforward. Any reasonable invoice-matching system can identify it.

But detecting the discrepancy raises a more difficult question: What should the finance team actually do about it?

Should the invoice be corrected? Rejected? Escalated? Has something similar happened before? And if it has, is that earlier decision still consistent with the company's current policy?

These questions led us to build LedgerMind, a policy-aware invoice exception resolution system that combines deterministic validation, persistent decision memory, and human oversight.

We started with a simple observation: an AI system that remembers earlier decisions is not necessarily an AI system that makes better decisions.

It also needs to know when those memories should not be trusted.

The Gap We Wanted to Address

Accounts payable teams commonly compare three documents:

Purchase Order (PO): What the company agreed to purchase.
Goods Receipt Note (GRN): What the company actually received.
Supplier Invoice: What the supplier wants to be paid.

This is known as three-way matching.

Matching can identify incorrect prices, quantity differences, missing receipts, duplicate invoices, and other exceptions.

Existing financial software already provides various matching and exception-management capabilities. Our aim wasn't to claim those systems cannot detect discrepancies.

We wanted to explore a different challenge: how can an AI assistant use an organization's previous resolution decisions without treating old behavior as unquestionable authority?

Historical decisions may be scattered across systems or attached to individual reviewers. Some may reflect outdated policies. Others may have been exceptional approvals that should never become general rules.

Our idea was to connect historical decision context with current policy and show the reasoning to a human reviewer.

What We Built

LedgerMind is a Python application with a Streamlit interface.

Its architecture brings together several components.

Python matching engine: Compares invoices, purchase orders, and goods receipts using deterministic rules. Financial calculations do not depend on a language model guessing the answer.

SQLite: Stores vendors, purchase orders, goods receipts, invoices, and human reviews. It provides a local business-record database separate from agent memory.

Hindsight: Provides persistent memory capabilities for retaining previous decisions and retrieving relevant historical context.

Policy engine Retrieves applicable written policies and checks whether a historical precedent is consistent with those rules.

Groq-hosted language model: Helps produce understandable explanations and proposed resolution actions.

Human review workflow: Allows a person to evaluate and record the final decision. LedgerMind does not automatically pay invoices.

The prototype includes synthetic historical decisions and test invoices. It also supports adding new invoices through manual entry and structured CSV intake.

We intentionally separated transaction storage from institutional memory.

SQLite answers, "What invoice did we receive?"

Hindsight helps answer, "What have we learned from similar decisions?"

They serve different purposes.

Why Hindsight Became Central to the Design

Most of the interesting questions appeared after the initial invoice matching was complete.

If an invoice contains an exception, we want the system to find relevant earlier resolutions.

For example, the agent could retrieve previous decisions involving a similar supplier, exception category, or pricing discrepancy.

With Hindsight, we developed a workflow around persistent experience.

The important concepts include:

Retain: Store useful information from reviewed outcomes.
Recall: Retrieve relevant past information when another case appears.
Reflect: Derive broader insights from accumulated experiences, where supported by the memory system.

In LedgerMind, retain and recall are particularly important. A reviewed outcome can become part of the decision history, while later investigations can use previously retained information as context.

The Hindsight documentation explains the memory system's capabilities. The broader motivation is also discussed in Vectorize's guide to agent memory.

But using memory exposed a subtle risk.

An agent can retrieve something that looks relevant and still misunderstand what that evidence means.

That problem shaped several of our most important engineering decisions.

Innovation One: Memory Does Not Override Policy

Consider our synthetic Prabhav Infosystems invoice scenario.

The purchase order specifies ₹1,800 per unit. The supplier charges ₹1,872 instead.

For 100 units with the scenario's 18% GST assumption:

Expected total: ₹212,400
Invoiced total: ₹220,896
Financial discrepancy: ₹8,496

A traditional rule-based matcher can detect this discrepancy.

LedgerMind goes further by combining the current policy with available historical resolution evidence.

In this scenario, the applicable policy supports correcting the invoice to the agreed purchase-order rate. The system proposes an edited payable amount of ₹212,400 for human review.

The critical principle is that a historical approval is evidence of what happened, not automatic permission to repeat it.

Even if an earlier reviewer accepted a similar increase, that decision should not silently override the policy applicable today.

We built policy–precedent conflict detection around this principle.

This is the part of LedgerMind we consider most interesting: memory supports decision-making, but current policy remains an independent constraint.

Innovation Two: A Memory Is Only as Reliable as Its Metadata

During development, we discovered a surprising bug.

A historical review from Zenith Precision Engineering appeared with metadata identifying it as a Prabhav Infosystems decision.

The original review content was not the problem. The issue came from how our application matched vendor names before storing the memory.

Common company-name suffixes such as "Pvt Ltd" caused unrelated vendors to be treated as possible matches.

That meant a correctly written review could still receive an incorrect vendor tag.

We changed the vendor identification logic to avoid relying on such weak matches.

The lesson was immediate: retrieval accuracy begins before retrieval.

If incorrect labels are attached when a memory is stored, even a good retrieval engine can return misleading evidence.

We also made it clearer when a recalled precedent belongs to a different vendor.

Cross-vendor evidence can be useful, but it shouldn't be presented as if it came from the same supplier.

Innovation Three: Five Facts Are Not Five Decisions

Another issue involved how memory information was displayed.

One historical review could produce several extracted facts:

The supplier name
The invoice discrepancy
The relevant policy
The reviewer action
The review rationale

Initially, those facts could appear as separate evidence cards.

This created a misleading impression that multiple independent decisions supported the same outcome.

We changed the interface to group related facts under their original review.

A single review now represents one logical historical precedent, with supporting details available inside it.

This matters because apparent evidence volume can affect how confident a proposed recommendation looks.

Five observations about one decision should not be interpreted as five independent decisions.

Innovation Four: An Invoice Must Not Validate Itself

One of our more subtle problems involved previous human feedback.

Suppose an invoice is reviewed, and its outcome is retained in memory.

When the same invoice is reopened, the memory system may retrieve that previous review.

That's useful for the audit trail.

But there is a problem if the agent treats its own earlier review of that invoice as independent evidence supporting a new recommendation.

The reasoning becomes circular.

We updated the resolution workflow to separate the current invoice's previous review from independent historical precedents.

The earlier review can still appear under the audit history, but it should not inflate precedent-based confidence.

This taught us that reliable memory needs boundaries, not just persistence.

What Happens With a Newly Created Invoice?

We also wanted LedgerMind to work beyond a fixed set of prepared examples.

In one user-created Nova Tech Supplies scenario, the purchase order specified ₹1,000 per unit, while the invoice charged ₹1,100.

For ten units with 18% GST:

Expected purchase-order total: ₹11,800
Supplier invoice total: ₹12,980
Difference: ₹1,180

LedgerMind identified the price mismatch using its deterministic matching logic.

The invoice record was saved in SQLite and could be accessed again after refreshing the application.

From there, the Resolution Workbench could present the discrepancy, applicable policy context, available historical evidence, and a proposed next action.

The final outcome remains subject to human review.

This distinction matters to us. We want the system to assist with investigation and consistency, not silently replace financial accountability.

What We Learned About Live Memory and Fallbacks

One practical challenge was making memory retrieval reliable.

Our implementation includes a local JSONL fallback for situations where live Hindsight retrieval is unavailable.

This keeps historical context accessible, but it also creates a responsibility to identify the actual source of that context.

During testing, Hindsight's server logs confirmed successful recall operations from our memory banks, including one request that returned 51 facts and another that returned nine.

We also encountered application requests where the interface used offline fallback.

Those two observations are not interchangeable.

A successful Hindsight server request does not automatically prove that a particular displayed recommendation used live retrieval.

That's why our interface distinguishes evidence attributed to the Hindsight vector database from evidence loaded through the offline JSONL store.

We learned that honest provenance is as important as the recommendation itself.

A reviewer needs to know not only what an agent remembers, but where that evidence came from.

What Comes Next

LedgerMind is currently a working prototype using synthetic business scenarios.

Its current capabilities demonstrate invoice intake, deterministic exception detection, local record persistence, policy-aware resolution proposals, historical memory integration, and human review.

Production deployment would require further work, including enterprise ERP integrations, stronger access control, monitoring, security reviews, and validation against real organizational processes.

Those limitations are important, but they don't change the central insight we gained from building the system.

We originally wanted to give an AI assistant memory so that it could learn from earlier financial decisions.

What we discovered was more interesting.

An AI agent shouldn't simply remember what happened. It should know where that memory came from, whether it applies to the present situation, and whether following it is still justified.

That's the principle behind LedgerMind.

DEV Community: Kiran Hooli