How I built **SentinelPay—a fault-tolerant double-entry ledger and stateful AI dispute resolver where the LLM scores risk, and plain code decides who gets their money back.
In banking software, two failure modes destroy trust faster than anything else: corrupting balances under concurrent retries, and letting a probabilistic AI model make deterministic financial decisions.
Most AI fintech demos wire an LLM straight to a refund endpoint and call it a day. I built SentinelPay to do the opposite—enforcing strict distributed-systems invariants at the ledger layer while keeping generative AI strictly inside an auditable, deterministic guardrail boundary.
Three Non-Negotiable Guarantees
Every architectural decision in SentinelPay flows from three invariants:
-
Money is never created, lost, or double-charged. Transfers execute inside a single PostgreSQL ACID transaction with deterministic row locks and two-layer
Idempotency-Keyenforcement. -
A retry or traffic burst cannot degrade the ledger. Per-account sliding-window rate limiting runs on Redis, and slow dispute analysis is decoupled via Redis Streams with retries and a dead-letter queue (
tx_dlq). -
An AI model never decides a refund. The LangGraph agent retrieves account history and asks the LLM for a structured risk score only. A unit-tested pure Python function makes the final credit-vs-escalation call, and the full reasoning trace is persisted in PostgreSQL
JSONB.
System Architecture
SentinelPay splits responsibilities across four containerized services (plus PostgreSQL 16 and Redis 7):
-
API Gateway (FastAPI
:8001): Verifies HS256 JWTs, enforces per-account Redis rate limits, checksIdempotency-Keyreplays, forwards transfers, and publishes events to Redis Streams. -
Ledger Service (FastAPI
:8002): The only service permitted to mutate balances. Executes atomic double-entry bookkeeping (DEBIT+CREDIT) with row-level locks. -
Agent Worker (LangGraph): Consumes
DISPUTE_OPENEDevents fromtx_eventsvia a Redis consumer group, runs a 3-node workflow, and persists resolutions. -
Console (Next.js 14 + TypeScript
:3000): Transfer simulator, duplicate-retry tester, live ledger journal, and agent reasoning inspector.
1. Preventing Deadlocks and Double-Spends in PostgreSQL
When two concurrent transfers hit the same pair of accounts in opposite directions (Alice → Ben and Ben → Alice), naive row locking causes deadlocks. And when five simultaneous $400 withdrawals hit a $1,000 balance, naive read-then-write logic allows a double-spend.
SentinelPay prevents both inside a single SQL transaction:
-
Integer cents only: Balances are stored as
BIGINTcents with a schema-levelCHECK (balance_cents >= 0)constraint—never floating-point numbers. - Deterministic lock ordering: Before mutating balances, the ledger locks both accounts in sorted ID order so opposite transfers always acquire locks in the exact same sequence:
SELECT account_id, balance_cents FROM accounts
WHERE account_id = ANY($1::uuid[]) ORDER BY account_id FOR UPDATE;
Two-Layer Idempotency
Network timeouts mean clients will retry POST /api/v1/transfers. Every transfer requires an Idempotency-Key header enforced at two layers:
- The API Gateway checks whether the key already exists and replays the original transaction with
200 OK(instead of201 Created). - If two identical requests arrive at the exact same instant and slip past the gateway check, the Ledger Service re-checks the key after acquiring row locks, backed by a
UNIQUEconstraint ontransactions.idempotency_keyas the final database backstop.
2. The Same-Millisecond Bug in Our Redis Rate Limiter
To stop burst traffic from hammering the ledger, each account has a Redis sorted set (ZSET) implementing a 60-second sliding-window log (max 10 requests/minute, returning 429 Too Many Requests with Retry-After: 60).
The bug: In version 1, I used the millisecond timestamp as both the sorted-set score and member. Under concurrent bursts, two requests arriving within the same millisecond overwrote each other's member in the set—making ZCARD undercount and silently letting excess traffic slip through.
The fix: Appending a short random nonce to each member makes same-millisecond requests distinct while keeping the timestamp as the score:
import uuid
pipe.zremrangebyscore(key, 0, now_ms - 60_000).zadd(key, {f"{now_ms}:{uuid.uuid4().hex[:8]}": now_ms}).zcard(key)
A regression unit test using an in-memory fake Redis now proves that 12 requests fired in the exact same millisecond yield 10 allowed and 2 blocked, whereas the original implementation allowed all 12.
3. Decoupling Slow AI From Fast Payments
LLM inference takes seconds; payment APIs must return in milliseconds. Whenever a user raises a dispute (or sends a transfer of $500+, which is auto-flagged), the gateway pushes an event to the Redis Stream tx_events and returns immediately.
The Agent Worker reads via a consumer group (XREADGROUP):
- Transient failures retry 3 times with exponential backoff (2s, 4s).
- Exhausted payloads are routed to a dead-letter stream (
tx_dlq) so no customer dispute silently vanishes. - Unacknowledged messages from a crashed worker are replayed on restart, and dispute resolution itself is idempotent—a redelivered event can never issue a second provisional credit.
4. Why the AI Never Decides a Refund
The dispute worker runs a 3-node LangGraph state machine:
-
retrieve_context: Loads the transaction, the sender's last 10 transfers, 30-day average spend, and prior dispute count from PostgreSQL. -
evaluate_policy: Computes numeric deviation in code and passes the customer's dispute reason as untrusted input to the LLM, which returns a strict Pydantic schema:{risk_score, category, explanation}. -
guardrails(pure deterministic code): Evaluates three hard, inclusive thresholds—every check must pass for an automatic provisional credit; otherwise it escalates to human operations:
def decide_dispute(amount_cents: int, risk_score: float, prior_disputes: int) -> str:
return "AUTO_CREDIT_APPROVED" if amount_cents <= 10_000 and risk_score <= 0.35 and prior_disputes == 0 else "ESCALATED_TO_OPS"
Real-World Proof: Catching an LLM Hallucination in Action
During live testing, I sent a $25 transfer seconds after two others—one of which was $499—and disputed it with "I did not authorise this payment."
-
Context: Amount
$25.00,0prior disputes, 30-day mean$262.00. -
LLM Output: Risk score
0.45, category velocity anomaly. -
Guardrails: Amount OK (
<= $100), prior disputes OK (0), risk0.45 > 0.35(FAIL). -
Decision:
ESCALATED_TO_OPS(DISPUTE_ESCALATED).
When I inspected the stored JSONB trace, the model's prose explanation falsely described the burst as "three transfers of $25"—completely hallucinating the $499 transfer. Yet the final decision was 100% correct, because the rule engine reads structured numbers, never prose. That single run validated the entire architecture.
5. Verification & Honest Limitations
All core guarantees are backed by an 11-test pytest suite (unit + live Docker integration):
| Invariant Tested | Result |
|---|---|
| Concurrency double-spend: 5 simultaneous $400 transfers on a $1,000 balance | 2 succeeded, 3 INSUFFICIENT_FUNDS, final balance $200.00
|
Idempotent retry: Same Idempotency-Key sent twice |
201 then 200 replayed; 1 transaction, 1 DEBIT + 1 CREDIT
|
| Burst rate limit: 12 requests from one account within 60s | Requests 1–10 → 201; 11–12 → 429
|
| Same-millisecond regression: 12 simultaneous timestamps | 10 allowed, 2 blocked |
| Guardrail boundary matrix: Inclusive limits & confidence bounds | 7/7 parameterized cases passed |
What I'd Build Next for Production
SentinelPay is a portfolio-scale architecture with deliberate simplifications I document upfront:
-
Bank suspense counter-entry: Today, a provisional credit adds a single
CREDITentry without debiting a bank suspense account. Production double-entry requires a counter-ledger entry. - Deterministic velocity features: Computing "transfers in the last N minutes" directly in SQL/code rather than letting the LLM infer bursts from raw timestamps.
-
Dead-letter visibility in UI: Surfacing
tx_dlqstatus in the console rather than leaving a failed dispute showing Agent reviewing.
Explore the Code
The complete Docker Compose setup, SQL schema, LangGraph worker, and pytest verification logs are open source on GitHub: github.com/raghuvendra34/sentinelpay
Top comments (0)