DEV Community

Raghuvendra Kumar
Raghuvendra Kumar

Posted on Edited on

Designing Idempotent Payment Microservices & Guardrailed LangGraph Dispute Agents

 How I built **SentinelPay—a fault-tolerant double-entry ledger and stateful AI dispute resolver where the LLM scores risk, and plain code decides who gets their money back.

In banking software, two failure modes destroy trust faster than anything else: corrupting balances under concurrent retries, and letting a probabilistic AI model make deterministic financial decisions.

Most AI fintech demos wire an LLM straight to a refund endpoint and call it a day. I built SentinelPay to do the opposite—enforcing strict distributed-systems invariants at the ledger layer while keeping generative AI strictly inside an auditable, deterministic guardrail boundary.

SentinelPay console: T-account balances and the live ledger journal


Three Non-Negotiable Guarantees

Every architectural decision in SentinelPay flows from three invariants:

  1. Money is never created, lost, or double-charged. Transfers execute inside a single PostgreSQL ACID transaction with deterministic row locks and two-layer Idempotency-Key enforcement.
  2. A retry or traffic burst cannot degrade the ledger. Per-account sliding-window rate limiting runs on Redis, and slow dispute analysis is decoupled via Redis Streams with retries and a dead-letter queue (tx_dlq).
  3. An AI model never decides a refund. The LangGraph agent retrieves account history and asks the LLM for a structured risk score only. A unit-tested pure Python function makes the final credit-vs-escalation call, and the full reasoning trace is persisted in PostgreSQL JSONB.

System Architecture

SentinelPay splits responsibilities across four containerized services (plus PostgreSQL 16 and Redis 7):

  • API Gateway (FastAPI :8001): Verifies HS256 JWTs, enforces per-account Redis rate limits, checks Idempotency-Key replays, forwards transfers, and publishes events to Redis Streams.
  • Ledger Service (FastAPI :8002): The only service permitted to mutate balances. Executes atomic double-entry bookkeeping (DEBIT + CREDIT) with row-level locks.
  • Agent Worker (LangGraph): Consumes DISPUTE_OPENED events from tx_events via a Redis consumer group, runs a 3-node workflow, and persists resolutions.
  • Console (Next.js 14 + TypeScript :3000): Transfer simulator, duplicate-retry tester, live ledger journal, and agent reasoning inspector.

1. Preventing Deadlocks and Double-Spends in PostgreSQL

When two concurrent transfers hit the same pair of accounts in opposite directions (Alice → Ben and Ben → Alice), naive row locking causes deadlocks. And when five simultaneous $400 withdrawals hit a $1,000 balance, naive read-then-write logic allows a double-spend.

SentinelPay prevents both inside a single SQL transaction:

  • Integer cents only: Balances are stored as BIGINT cents with a schema-level CHECK (balance_cents >= 0) constraint—never floating-point numbers.
  • Deterministic lock ordering: Before mutating balances, the ledger locks both accounts in sorted ID order so opposite transfers always acquire locks in the exact same sequence:
SELECT account_id, balance_cents FROM accounts
WHERE account_id = ANY($1::uuid[]) ORDER BY account_id FOR UPDATE;
Enter fullscreen mode Exit fullscreen mode

Two-Layer Idempotency

Network timeouts mean clients will retry POST /api/v1/transfers. Every transfer requires an Idempotency-Key header enforced at two layers:

  1. The API Gateway checks whether the key already exists and replays the original transaction with 200 OK (instead of 201 Created).
  2. If two identical requests arrive at the exact same instant and slip past the gateway check, the Ledger Service re-checks the key after acquiring row locks, backed by a UNIQUE constraint on transactions.idempotency_key as the final database backstop.

2. The Same-Millisecond Bug in Our Redis Rate Limiter

To stop burst traffic from hammering the ledger, each account has a Redis sorted set (ZSET) implementing a 60-second sliding-window log (max 10 requests/minute, returning 429 Too Many Requests with Retry-After: 60).

The bug: In version 1, I used the millisecond timestamp as both the sorted-set score and member. Under concurrent bursts, two requests arriving within the same millisecond overwrote each other's member in the set—making ZCARD undercount and silently letting excess traffic slip through.

The fix: Appending a short random nonce to each member makes same-millisecond requests distinct while keeping the timestamp as the score:

import uuid
pipe.zremrangebyscore(key, 0, now_ms - 60_000).zadd(key, {f"{now_ms}:{uuid.uuid4().hex[:8]}": now_ms}).zcard(key)
Enter fullscreen mode Exit fullscreen mode

A regression unit test using an in-memory fake Redis now proves that 12 requests fired in the exact same millisecond yield 10 allowed and 2 blocked, whereas the original implementation allowed all 12.


3. Decoupling Slow AI From Fast Payments

LLM inference takes seconds; payment APIs must return in milliseconds. Whenever a user raises a dispute (or sends a transfer of $500+, which is auto-flagged), the gateway pushes an event to the Redis Stream tx_events and returns immediately.

The Agent Worker reads via a consumer group (XREADGROUP):

  • Transient failures retry 3 times with exponential backoff (2s, 4s).
  • Exhausted payloads are routed to a dead-letter stream (tx_dlq) so no customer dispute silently vanishes.
  • Unacknowledged messages from a crashed worker are replayed on restart, and dispute resolution itself is idempotent—a redelivered event can never issue a second provisional credit.

4. Why the AI Never Decides a Refund

The dispute worker runs a 3-node LangGraph state machine:

  1. retrieve_context: Loads the transaction, the sender's last 10 transfers, 30-day average spend, and prior dispute count from PostgreSQL.
  2. evaluate_policy: Computes numeric deviation in code and passes the customer's dispute reason as untrusted input to the LLM, which returns a strict Pydantic schema: {risk_score, category, explanation}.
  3. guardrails (pure deterministic code): Evaluates three hard, inclusive thresholds—every check must pass for an automatic provisional credit; otherwise it escalates to human operations:
def decide_dispute(amount_cents: int, risk_score: float, prior_disputes: int) -> str:
    return "AUTO_CREDIT_APPROVED" if amount_cents <= 10_000 and risk_score <= 0.35 and prior_disputes == 0 else "ESCALATED_TO_OPS"
Enter fullscreen mode Exit fullscreen mode

Agent reasoning inspector showing risk score, guardrail checks, and LangGraph node traces

Real-World Proof: Catching an LLM Hallucination in Action

During live testing, I sent a $25 transfer seconds after two others—one of which was $499—and disputed it with "I did not authorise this payment."

  • Context: Amount $25.00, 0 prior disputes, 30-day mean $262.00.
  • LLM Output: Risk score 0.45, category velocity anomaly.
  • Guardrails: Amount OK (<= $100), prior disputes OK (0), risk 0.45 > 0.35 (FAIL).
  • Decision: ESCALATED_TO_OPS (DISPUTE_ESCALATED).

When I inspected the stored JSONB trace, the model's prose explanation falsely described the burst as "three transfers of $25"—completely hallucinating the $499 transfer. Yet the final decision was 100% correct, because the rule engine reads structured numbers, never prose. That single run validated the entire architecture.


5. Verification & Honest Limitations

All core guarantees are backed by an 11-test pytest suite (unit + live Docker integration):

Invariant Tested Result
Concurrency double-spend: 5 simultaneous $400 transfers on a $1,000 balance 2 succeeded, 3 INSUFFICIENT_FUNDS, final balance $200.00
Idempotent retry: Same Idempotency-Key sent twice 201 then 200 replayed; 1 transaction, 1 DEBIT + 1 CREDIT
Burst rate limit: 12 requests from one account within 60s Requests 1–10 → 201; 11–12 → 429
Same-millisecond regression: 12 simultaneous timestamps 10 allowed, 2 blocked
Guardrail boundary matrix: Inclusive limits & confidence bounds 7/7 parameterized cases passed

What I'd Build Next for Production

SentinelPay is a portfolio-scale architecture with deliberate simplifications I document upfront:

  • Bank suspense counter-entry: Today, a provisional credit adds a single CREDIT entry without debiting a bank suspense account. Production double-entry requires a counter-ledger entry.
  • Deterministic velocity features: Computing "transfers in the last N minutes" directly in SQL/code rather than letting the LLM infer bursts from raw timestamps.
  • Dead-letter visibility in UI: Surfacing tx_dlq status in the console rather than leaving a failed dispute showing Agent reviewing.

Explore the Code

The complete Docker Compose setup, SQL schema, LangGraph worker, and pytest verification logs are open source on GitHub: github.com/raghuvendra34/sentinelpay

Top comments (0)