DEV Community: Someswararao Tellakula

I Gave My Accounts-Payable Agent Hindsight Memory. It Stopped Asking.

Someswararao Tellakula — Sun, 08 Nov 2026 03:52:46 +0000

Every month, an invoice from a steel distributor arrives priced 3% above the purchase order, and every month someone in accounts payable stops, opens the contract, finds clause 7.2 (monthly steel-index adjustment, capped at 5%), and approves it. The matching software flags it perfectly. It just has no idea that this exact question was answered last month, and the month before.

That is the gap I built Ledgerwise for: an agent that sits after the 3-way match and resolves invoice exceptions the way your desk resolved them last time — and knows when that history isn't enough.

What the system does

An AP desk matches three documents: the purchase order (what we agreed to buy), the goods receipt (what actually arrived), and the invoice (what the vendor wants paid). When they disagree you get an exception: a price variance, a short shipment, a GST rate that doesn't match the contract, a surcharge line nobody ordered, a re-sent duplicate, or a vendor letter saying "we've changed our bank account."

Detecting those is a solved problem, and in Ledgerwise it's deliberately boring: matching.py is deterministic Python with no model and no memory. It finds exceptions and measures them — a percentage for price, rupees for a surcharge, units for a short shipment.

The interesting part is deciding. That's agent.py, and it runs one loop per exception:

Recall how this vendor's same kind of exception was resolved before.
Apply a conservative precedent policy.
Auto-resolve if human precedent is consistent; otherwise escalate with the precedents attached.
Retain every human decision, including the reason they typed, so the next one is easier.

The memory layer is Hindsight, an open-source agent memory system from Vectorize. I picked it over rolling my own vector store because AP history isn't a pile of documents to search — it's a set of decisions with actors, amounts and dates, and I wanted something that extracts and organises those facts rather than just embedding paragraphs.

The core story: memory is only useful if you know when not to trust it

The obvious design is: recall similar past decisions, and if the top result says APPROVE, approve. Run that against the replay data and it looks great for a few weeks — until a logistics vendor sends a ₹6,800 fuel surcharge. Every earlier surcharge from that vendor was approved, but they were ₹1,200–₹2,400, and the agreed cap is ₹2,500. Similarity says "same vendor, same exception, approved." The money says otherwise.

That case reshaped the whole design. The question stopped being "what does memory say?" and became "is memory enough here?"

Writing memories the policy can actually use

Each human decision is retained as one string with two parts: a prose sentence that Hindsight's retain pipeline extracts facts and entities from, and a structured line my policy can parse back out.

def retain_decision(self, invoice, exc, decision, reason, decided_by):
    vendor = self.vendors[invoice["vendor_id"]]
    self.memory.retain(
        precedent_text(vendor, exc["type"], exc["magnitude"], decision,
                       invoice["id"], decided_by, reason),
        context=f"accounts payable exception resolution — {vendor['category'].lower()} vendor",
        document_id=f"{invoice['id']}:{exc['type']}",
        metadata={"vendor_id": vendor["id"], "exception": exc["type"],
                  "decision": decision, "decided_by": decided_by},
        timestamp=invoice["date"],
    )

document_id keeps a re-decision on the same invoice from becoming a duplicate memory, and timestamp uses the invoice date rather than "now", so temporal questions work.

The magnitude matters as much as the decision. "Approved a 2.8% variance" and "approved a 4.2% variance" are different precedents.

Recalling with the source attached

Before touching an exception, the agent asks a question in plain English:

query = (f"How were {exc.type.replace('_', ' ').lower()} exceptions on invoices from "
         f"{vendor['name']} resolved, and why?")
resp = client.recall(bank_id=self.bank_id, query=query,
                     types=["world", "experience", "observation"],
                     budget="mid", max_tokens=4096,
                     include_chunks=True, max_chunk_tokens=600)

include_chunks=True was the detail that made this work. Recall returns the extracted facts, which are great for humans and for reflection, but the policy needs exact numbers. Asking for source chunks brings the structured precedent line back alongside each fact.

The policy

With precedents in hand, the rules are short and intentionally conservative:

if len(approved) >= MIN_PRECEDENTS and exc.magnitude <= cap and (floor is None or exc.magnitude < floor):
    return ExceptionDecision(exc.to_dict(), "APPROVE", True,
        f"{len(approved)} human approvals up to {max(approved):g}; this is {exc.magnitude:g}.", cited)

At least two human precedents before acting.
For amounts: approve only inside the range humans approved (plus 10% headroom) and below anything they rejected. Outside that range, escalate with a lean — "close to past approvals" or "far from anything approved".
For categorical exceptions (short shipment, GST mismatch, duplicate): act only if the last three human decisions agree.
Bank-account changes always go to a human. No amount of history makes a changed remit-to account safe to pay automatically.
The agent never learns from itself. Its own decisions are retained for audit, but tagged and excluded from precedent. Without that, one bad call becomes two precedents by Thursday.

The LLM (Groq, openai/gpt-oss-120b) writes the explanation and the vendor email. It never makes the pay/no-pay decision, and if it's down, a template takes over.

What it looks like over eight weeks

I replay eight weeks of invoices from six vendors, each with one realistic recurring quirk, and play the AP lead whenever the agent escalates.

	W1	W2	W3	W4	W5	W6	W7	W8	Overall
Without memory	0%	0%	0%	0%	0%	0%	0%	0%	0%
With Hindsight	0%	0%	75%	75%	60%	60%	100%	67%	56%

That's the share of exception invoices resolved with no human, on the replay dataset. Across the run, zero of the automatic decisions were ones the AP lead would have overturned.

The dips are the part I'm proudest of:

Week 1, Kaveri invoices 18% GST on a 12% line. No memory, so it comes to me. I reject and type: "HSN 2839 is contracted at 12% GST; ask Kaveri for a credit note. Never pay the 18% version."

Week 3, same quirk. Recall returns two human rejections. The agent rejects on its own, cites both invoice numbers, and drafts the credit-note email. Nobody touches it.

Week 5, steel at +3.1%. Humans had approved up to +2.8%. The agent escalates with "leaning approve" instead of extrapolating. I approve, and the learned range widens.

Week 6, Meridian Electricals sends a "we've moved banks" letter. Memory has six clean Meridian invoices. It escalates anyway.

The "Brief" button calls Hindsight's reflect for a vendor and returns what a new joiner should know before the payment run: fuel surcharges up to ₹2,500 are pre-agreed, one at ₹6,800 was rejected. That's the institutional knowledge that usually leaves when the AP lead goes on leave.

See it run

Youtube link: https://youtu.be/3vHxH_v8v6

Try the live demo: https://ledgerwise-1ydb.onrender.com/

Lessons

1. The reason field is the product. I made the UI refuse to save a decision without a typed reason. Those sentences are what escalations show, what reflect summarises, and what makes the memory auditable. "Approved" is a log line; "approved under clause 7.2, index sheet attached" is knowledge.

2. Store the magnitude, not just the outcome. Similarity search will happily match "same vendor, same exception" and ignore that this one is three times bigger. Every precedent carries its number, and the policy compares numbers.

3. Decide what memory is allowed to override before you write any recall code. Bank changes, unknown POs and anything outside a learned range are human-only by design. Writing those guardrails first made the rest of the policy much easier to reason about.

4. Don't let the agent cite itself. Retaining the agent's own decisions is useful for audit and reflection. Counting them as precedent is how a system drifts.

5. Make memory configurable, not magical. A skeptical, literal bank disposition (skepticism 5, literalism 4) fit an AP desk better than the defaults. One approval of a 3% variance should not become permission for "price variances" in general.

The honest limitation: Ledgerwise is only as good as the reasons people type. A desk that clicks "approve" with "ok" gets an agent that can count but can't explain.

If you want to try the memory layer yourself, the Hindsight docs get you to a working retain/recall loop quickly, and Vectorize's explainer on what agent memory is is a good read on why this is different from RAG. Ledgerwise's code is on GitHub: https://github.com/SomeswararaoTellakula/ledgerwise

Shout-out to Code.in and the Hindsight team at Vectorize.

Hindsight taught my on-call agent which fixes backfire

Someswararao Tellakula — Tue, 29 Sep 2026 16:29:38 +0000

Hindsight taught my on-call agent which fixes backfire

At 11:06 on a Monday, checkout starts failing. The alert says HikariPool-1 - Connection is not available, request timed out after 30000ms. I gave that alert to gpt-oss-120b with no history. Its first recommendation: raise the connection pool from 40 to 80 per pod and restart the pods.

At this company, that is how you cause an outage. Five months earlier, a restart bought four minutes before the pool exhausted again, and scaling out blew through the database's 500-connection limit and took payments down with it. Twelve pods at 80 connections is 960. It's all in a postmortem that nobody reads at 11:06 with a SEV1 going off.

So I built NeverTwice, an on-call copilot that reads the postmortems for you, using Hindsight as its memory. This post is about the design decision that mattered most: remembering what didn't work as carefully as what did — and then making memory enforce it.

The setup

NeverTwice triages alerts for BasketBolt, a fictional 10-minute grocery delivery company in Hyderabad. I wrote six months of history for it: 15 postmortems across checkout (Spring Boot + HikariCP), payments (UPI through two payment providers), Kafka consumers, Redis, OpenSearch, SMS OTPs and Kubernetes. Each has real-shaped logs, a timeline, a root cause, open action items and every remediation attempt with its outcome: worked, didn't work, made it worse.

The stack is small: Python/Starlette, a vanilla JS UI, Groq's gpt-oss-120b, and Hindsight for everything the agent knows. Hindsight extracts facts and entities from what you retain, searches them four ways at once on recall (semantic, keyword, entity graph, time), consolidates repeated facts into observations, and answers questions with reflect. Loading the 63 memory items took about a minute of extraction.

Decision 1: store outcomes as separate, tagged memories

If a postmortem is retained as one document, the restart that failed and the fix that worked sit in the same blob, and "restart" is such a natural first step that a model can hand it straight back. So each incident is split by outcome:

items.append({
    "content": "\n".join(lines),   # "At 11:15 IST the team tried: Rolling restart ... Outcome: DID NOT WORK. ..."
    "context": f"{iid} remediation — {OUTCOME_LABEL[outcome]} ({svc})",
    "timestamp": ts,
    "document_id": f"{iid}:remediation:{outcome}",
    "tags": base_tags + ["kind:remediation", f"outcome:{outcome}"],
    "metadata": {"incident_id": iid, "service": svc, "kind": "remediation", "outcome": outcome},
})

For every alert, NeverTwice runs four recalls in parallel — similar incidents, fixes that worked, fixes that failed or backfired, and on-call rules — and the outcome recalls are scoped by tag:

("failed", "Recalling fixes that failed",
 dict(query=f"remediation that failed or made things worse for {symptom}", budget="low", max_tokens=900,
      tags=["outcome:failed", "outcome:made_worse"], tags_match="any", types=["world", "experience"])),

Results are numbered M1..Mn, and the model must cite them or incident IDs as evidence.

Before and after

Same model, same prompt, same alert; the only difference is memory.

Without memory, the model recommended raising the HikariCP pool to 80, restarting the pods, and said it had "seen this before" in INC-2025 — an incident that doesn't exist. NeverTwice checks every cited ID against what recall actually returned, so that chip shows up red as unverified.

With Hindsight, it recalled 28 memories and cited four real incidents (INC-2104, INC-2291, INC-2317, INC-2282). Its first step was a verification query, not a mitigation. It said to cut the new reconcile workers back and terminate idle-in-transaction sessions, and under Don't do this it listed scaling checkout to 20 replicas (INC-2104) and changing the database instance mid-incident (INC-2282).

[Screenshot: the Compare view]

The dead end: memory that informs isn't enough

Then I read the memory answer closely. Step 4 was: "If latency remains high, perform a rolling restart of checkout-api pods." Memory had recalled that restarts relapsed in two incidents. The model even cited INC-2317 as support — but the fix there was a rollback, not a restart. Recall worked; the model still ignored it.

Prompting harder helps a little. What fixed it was treating memory as a check, not just context. After the model answers, a backfire guard looks at every risky step — restarts, scaling out, raising timeouts, database limit changes — and asks Hindsight directly whether that kind of fix failed before for the same service:

query = f"{name} of {service} did not work or made things worse"
for group in ("failed", "worked"):
    for raw in await self.memory.search(query, group):
        ...
moved = backfire_guard(report, MemoryContext(items=ctx.items + extra), service)

If failures outnumber successes, the step moves to Don't do this with the proof and a "caught by guard" badge. The "outnumber" rule matters: in one incident a restart was the right fix (a rotated secret is only read at startup), so the guard can't be a blanket ban.

[Screenshot: the "caught by guard" card]

Decision 2: let it learn in front of you

One alert is a failure BasketBolt has never seen: Telugu and Hindi searches return nothing because new OpenSearch nodes were built without the analysis-icu plugin. NeverTwice has nothing relevant to recall and says so, with low confidence.

The responder clicks Resolve & teach, records what worked, what didn't, and whether NeverTwice's advice was right. That becomes new memories — including an experience fact about NeverTwice's own advice. When the same error hits a new node five days later, it recalls the incident it was just taught and gives the exact fix.

Hindsight also maintains four mental models that act as living runbooks — Recurring failure patterns, Fixes that backfired, Open action items, On-call rules — refreshed automatically after consolidation. Three directives keep reflect honest: cite incident IDs, flag destructive actions, separate what worked from what backfired.

Measuring it

I didn't want "it feels smarter", so each alert has a checklist: steps it must recommend, known-bad steps it must not, incidents it should cite. A script triages each alert three times with and without memory and grades every answer.

On the payments alert, memory passed 3/3. Without memory, the model recommended raising the payment-provider timeout in 3/3 runs — the exact change that turned a degraded provider into an outage in INC-2150. On the checkout alert, the no-memory answer recommended a bigger connection pool 3/3 times.

Across all four alerts, three runs each:

	Passed the checklist	Recommended a fix that already failed
With Hindsight memory	11 / 12	1 / 12
Without memory	2 / 12	10 / 12

Without memory, the model's instincts were consistent and wrong: bigger connection pools, longer timeouts, more replicas, reset the consumer offsets. It never once found the dead-letter-queue fix for the crash-looping Kafka consumers, because that fix only exists in this team's history.

The one memory miss is the interesting one. On the DNS alert, it suggested scaling CoreDNS "if the DaemonSet is already present". Memory holds two answers to that: scaling CoreDNS fixed INC-2160 and failed in INC-2305, where the real cause was a missing node-local DNS cache. With one success and one failure, the guard correctly doesn't veto it; the model has to read why each one happened. That is where I'd push next: remembering the conditions under which a fix works, not just its outcome.

My first checklist was wrong, too: it failed two good answers because "scale the reconcile workers down to 4" matched my "no scaling" rule. Grading your grader is part of the job.

What didn't work, and what I'd change

Tool calling breaks in new ways. On the first live run, gpt-oss-120b called a tool named json that didn't exist, and Groq rejected the call. The model's arguments were fine, just under the wrong name, so NeverTwice now salvages them from the failed generation instead of paying for another call.
Free-tier limits shaped the design. Groq's free tier allows 8K tokens per minute on gpt-oss-120b, and a side-by-side comparison makes two calls. Four small tag-scoped recalls plus runbook excerpts keep a prompt around 3K tokens.
Keyword grading is crude. It undercounts answers phrased differently; an LLM judge would be fairer. I kept keywords because they're deterministic and I can read exactly why an answer failed.
The data is synthetic. The next step is importing real postmortems with the same schema.

Try it

The code, data and scripts are on GitHub: https://github.com/SomeswararaoTellakula/nevertwice. You need a Hindsight Cloud key (or a local Hindsight container) and a Groq key; ./start.sh sets everything up and starts the app.

The most valuable thing it did wasn't finding the right fix. It was stopping the wrong one.

Links: Hindsight on GitHub · Hindsight docs · What is agent memory?