My AP agent only trusts what Hindsight remembers

Trinu — Sat, 07 Nov 2026 16:33:34 +0000

An invoice from our packaging supplier arrived 4.52% above the purchase-order price, with a note: "revised rates due to kraft paper increase." The right call was to hold it. Their contract allows a surcharge of up to 2.5%, and nothing more. My agent got that right for one reason only: it remembered a single decision from weeks earlier, when a 4.26% variance from the same vendor was sent back to procurement. When that one memory went missing, the agent stopped being sure and handed the invoice to a person.

That behaviour is the whole design. I built Precedent, an accounts-payable exception agent, around one rule: the agent may only be as confident as its memory allows. The memory is Hindsight, an open-source agent memory system, and it turned out to be the most important part of the stack.

The problem: the same exceptions, every month

Every mid-size company pays vendors through a 3-way match: purchase order, goods receipt, invoice. The ones that don't match land on an AP executive's desk. What surprised me is how repetitive they are:

The packaging supplier adds a diesel surcharge, allowed up to 2.5% by clause 7.2 of their contract.
The grain trader's ERP bills in quintals against POs written in kilograms.
The logistics vendor invoices the full PO on the first partial dispatch.
The label printer re-sends unpaid invoices with /R on the end.
Someone sends a polite letter: "our bank account has changed, please remit to the new one."

The right call is obvious if you remember how it went last time. Usually exactly one senior person does. A stateless LLM doesn't help: give it the invoice, the PO and the policy, and it writes a thoughtful paragraph and escalates everything. Clause 7.2 isn't in the documents. It's in the history of decisions.

How it hangs together

![Precedent architecture: the agent recalls precedents from Hindsight, decides, and every outcome is retained back into memory]

A deterministic 3-way match flags the exception: price variance, quantity, unit of measure, GST type, unplanned charges, duplicates, bank-detail changes. The agent then does three things:

Recall precedents from Hindsight, scoped by vendor and exception type.
Decide: an LLM (Groq, gpt-oss-120b) reasons over those precedents and submits a decision through a function call. An auditable reasoner checks that decision against the evidence.
Retain: whatever happens, the outcome goes back into Hindsight. That includes a person overriding the agent, with their reason.

Above 75% confidence, the agent clears the invoice. Below that, a person confirms with one click. Bank-account changes always go to treasury, whatever the confidence.

Tags are the schema

Hindsight's retain, recall and reflect operations are simple, but the real design work was deciding what a memory is. Hindsight extracts facts from natural language, which is great for reflection. I also needed to recall precisely, so every resolved exception is written as prose and tagged like a database row:

"content": f"AP exception resolved on {inv['received_at']}. Vendor {inv['vendor_name']} ... "
           f"Exception: {primary.label} — {primary.summary} "
           f"Decision: {ACTIONS[final_action]} by {decided_by}. Reason: {reason}",
"tags": [f"vendor:{inv['vendor_id']}", f"exception:{primary.type}", f"action:{final_action}",
         f"mag:{mag}", f"outcome:{outcome}", "kind:case"],
"document_id": f"case-{inv['invoice_id']}",

The prose lets Hindsight build good facts and observations: "Shree Balaji's surcharges are approved under clause 7.2". The tags let the agent ask exact questions: this vendor, this exception, what was decided. The document_id matters more than I expected. Hindsight extracts several facts per document, so without de-duplicating by document, one case would vote three times.

Confidence comes only from memory

This is the part I'm most opinionated about. With an empty memory bank, every exception scores 0% and goes to a person. Autonomy is earned one resolved case at a time.

For price variances and extra charges, the reasoner learns the vendor's tolerance as a threshold rather than a vote. Approvals set a ceiling, holds set a floor, and the gap between them is a grey zone that always goes to a human:

if ok and m <= a_max + 1e-9 and (h_min is None or m < h_min):
    action, agree, sup = _vote(ok)
    conf = agree * min(1.0, len(ok) / 3)

The LLM's job is to explain and to weigh context the tags can't capture, like a vendor email or a fraud advisory. It isn't allowed to be more certain than the evidence:

if out["action"] == proposal.action and not no_backing:
    d.confidence = round(min(llm_conf, proposal.confidence), 2)
elif no_backing:
    # Memory has no precedent that covers this case: the LLM may suggest, not act alone.
    d.confidence = round(min(llm_conf, 0.4), 2)

If the model disagrees with what memory says, confidence is capped at 50% and a person decides. In practice this one min() is what makes the system safe to let run.

Before and after

I replayed 11 weeks of AP history (96 invoices across 12 vendors, modelled on a mid-size food company in Hyderabad) and let the agent recommend before each recorded human decision.

Week 1: 0% of recommendations matched what the team decided. It knew nothing, and said so.
Last 4 weeks: 100% matched, and 88% of exceptions were cleared without a person touching them.
Wrong auto-clears: zero, across the whole run.

The same invoices with memory switched off: 0 precedents, 0% confidence, everything escalated.

Hindsight recalled three earlier bank-change cases across other vendors (two were confirmed fraud) plus an internal advisory email about fake bank letters. Payment was blocked and sent to a treasury call-back. On top of that, a Hindsight directive makes "never pay a changed bank account without a call-back" a rule the memory layer itself enforces during reflection.

Hindsight's reflect also writes a vendor playbook from everything it remembers, with a JSON response schema so the UI can render it as a table. That's the onboarding document a new AP hire would otherwise piece together over weeks.

The bug that taught me the most

The first time I ran against Hindsight Cloud, the 4.52% invoice from the opening escalated instead of being held. In my local tests it had worked.

The cause: that vendor has eleven approvals and one hold. Hindsight ranks results by relevance within a token budget, and eleven near-identical approvals crowded out the single hold. Without it, the reasoner saw a ceiling of 2.43%, no floor, and decided 4.52% was "above anything seen before."

The fix was to ask for the boundary explicitly:

if exc.type in MAGNITUDE_TYPES:
    for action in BOUNDARY_ACTIONS:   # hold_for_procurement, return_to_vendor
        batches.append(self.memory.recall(
            f"{inv['vendor_name']} {exc.label} {ACTIONS[action]}: amount above the accepted limit",
            tags=[v, t, f"action:{action}"], tags_match="all_strict", limit=10))

Rare outcomes are often the ones that define the rule. If your agent reasons over thresholds, don't trust a single relevance-ranked query to surface them.

Lessons I'd reuse

Make memory the source of confidence, not a garnish. If an agent can be just as confident with an empty memory bank, the memory isn't doing anything.
Write memories for two readers. Prose for the memory system's fact extraction and reflection; tags for exact recall.
Retain the overrides, with the reason in the reviewer's words. They're the most valuable memories in the bank.
Query for the boundary explicitly. The case that defines a limit is usually the rarest one.
Function calling fails; plan for it. Groq models sometimes return tool_use_failed or malformed JSON. I re-prompt, repair the arguments, switch to a fallback model, and if all of that fails, decide from memory evidence alone with a visible warning.

One more, from the plumbing: the Hindsight Python SDK runs on aiohttp, and its sync wrappers tie the HTTP session to the first event loop that uses it. A Flask server answers each request on a different thread, so I run every SDK call on one dedicated event loop. It took far longer to find than to fix.

If you're building agents that make the same kinds of decisions over and over, agent memory is what turns "it gave a reasonable answer" into "it did what we always do, and here's the case where we did it."
Precedent's code is on GitHub: https://github.com/Sowmya-Krishna05/precedent

DEV Community: Trinu