Building a Chargeback Defense Agent That Isn't Allowed to Make Things Up

shivansh sinha — Sat, 07 Nov 2026 18:17:04 +0000

What Recoup actually does

A seller connects their Stripe account. When a dispute comes in, Recoup pulls the order record, shipping and delivery data, customer messages, and the seller's own policies, and drafts a response. It also tells the seller whether fighting is worth it at all, based on evidence strength and that seller's own track record with disputes like this one.

The part we spent the most time on isn't the drafting. It's making sure the agent can learn from experience without letting that experience override what's actually true about the dispute in front of it.

Splitting the system in two

We separated Recoup into a core application layer and an agent intelligence layer, talking to each other through a fixed API contract.

The core layer owns facts: Stripe webhook ingestion, the database, evidence collection, submission, and deadline tracking. The agent layer owns judgment: classification, evidence checklists, claim validation, decision scoring, response drafting, and memory.

That split exists for one reason: we didn't want the part of the system that reasons about a dispute to also be the part that decides what counts as evidence. Those need different levels of trust.

python
class EvidenceCollector:
def collect(self, dispute: Dispute) -> list[Evidence]:
return [
*self.order_service.get_receipts(dispute.order_id),
*self.shipping_service.get_delivery_records(dispute.order_id),
*self.messages_service.get_customer_history(dispute.customer_id),
]

Evidence collected this way is marked verified. Anything a seller uploads or types themselves is stored separately, as a claim, until something verifiable backs it up. That distinction runs through the entire system.

Every item on that checklist traces back to a real source — a shop API or a shipping API — with a timestamp. Nothing on this screen was typed in by a person.

Giving the agent memory that can be wrong about something small, without being wrong about everything

Most "AI with memory" we'd seen stores conversation history and retrieves it by similarity. That tells you what was said before, not what actually worked.

We used Hindsight, an open-source memory layer built for agents, to store something closer to experience: past dispute cases, which evidence won and which didn't, seller-specific operating rules, and beliefs that update as outcomes come in.

python
hindsight.retain(
bank=f"seller-{seller.id}",
content={
"dispute_id": dispute.id,
"reason_code": "item_not_received",
"evidence_used": ["delivery_photo", "tracking_number"],
"outcome": "won",
},
tags=["reason:not_received", "courier:bluedart", "outcome:won"],
)

Say a seller has won most of their "item not received" disputes when a delivery photo was attached. The next time a similar dispute comes in:

python
similar = hindsight.recall(
bank=f"seller-{seller.id}",
query="item not received, courier BlueDart",
tags=["reason:not_received"],
)

That result becomes a concrete recommendation, backed by a number, not a vibe. In practice it looks like this once a seller has enough history:

"Delivery photo wins 'not received' disputes, 62% with it versus 21% without, confidence 99%, n=30." That n=30 matters as much as the number itself — we show the sample size next to every belief, because a pattern from three cases isn't a pattern, it's a guess. The same page surfaces risky couriers and repeat disputers, and lets a seller add standing rules ("never fight disputes under 20") that apply to every future draft, no matter what memory says.

Not letting the model write checks it can't cash

Once we had evidence and memory, the easy part would have been letting an LLM write a confident-sounding rebuttal letter. We didn't want confident. We wanted correct.

Every claim in a generated response has to point at a specific evidence ID. That evidence has to belong to the dispute being handled and has to be verified. Unsupported claims get blocked before the letter ever reaches the seller. Once a seller approves a draft, we hash it, so what gets submitted is provably what they reviewed, not something regenerated after the fact.

This is the thing we think gets skipped too often in applied agent work: explainability and evidence integrity aren't a UI feature you bolt on. They're a constraint on the generation pipeline itself, or they don't mean anything.

Closing the loop on outcomes

A dispute resolves, weeks later, as won or lost. That outcome gets retained back into the seller's memory bank, and Hindsight's reflection step updates the relevant beliefs. That's the actual loop:

dispute → evidence → decision → response → outcome → memory → better next decision

Zoomed out across a seller's full history, that loop is what the analytics layer is actually showing:

None of these numbers come from the LLM guessing at a summary. They're computed directly from closed disputes, and every rate on the page states how many cases it's based on, for the same reason the Playbook does: a win rate means nothing without knowing the sample behind it.

What we learned

Facts and claims need to be different types from the start. We didn't bolt this on later, if a seller's unverified note and a verified shipping record live in t``he same table with the same trust level, you will eventually generate a letter that cites something that isn't real.

Fast signals matter more than slow ones. Dispute outcomes take weeks. A seller editing a draft or overriding a recommendation is immediate, and it happens far more often. We learn from both, but we didn't want the system sitting idle for a month waiting on the slow signal.

Idempotency is not optional with financial webhooks. Stripe will redeliver events. If a duplicate webhook creates a duplicate memory write, your agent "learns" the same lesson twice and its confidence numbers lie to you.

A belief needs a minimum sample size before it's allowed to speak. One win doesn't make a pattern. We gate recommendations on recurrence, and we show confidence and sample size explicitly rather than letting a thin pattern sound as certain as a strong one.

Separating what reasons from what's true is the real architecture decision. Everything else, which framework, which database, which LLM, was replaceable. The boundary between the agent's judgment and the system's facts was not.

Where this goes next

We're working on deeper evaluation of how memory actually changes recommendation quality over time, rather than just showing that it changes, and on supporting more payment processors beyond Stripe. The harder, more interesting problem is still ahead of us: measuring, honestly, how much a seller's own history should be allowed to influence a decision versus how much is just noise from too few cases.

If you're building agents that are supposed to improve with experience rather than with longer prompts, agent memory is worth taking seriously as its own architectural layer, not a feature you add to a vector store at the end.

Repository: https://github.com/shivansh-source/recoup
Stack: Python, FastAPI, Next.js, PostgreSQL, Hindsight, Docker, Stripe``

DEV Community: shivansh sinha