DEV Community: Pablets

The Saga Pattern Under an AI Agent — Orchestration, Idempotency, and the Transactional Outbox

Pablets — Tue, 28 Jul 2026 01:02:47 +0000

A loan journey is a saga: a sequence of commands across services, each individually transactional, none jointly so. Here's how an orchestrated saga stays correct when the network eats a response — retries that can't double-write, an outbox that can't lose an event, and a consumer that can't apply one twice.

There is exactly one failure that separates toy distributed systems from production ones, and it fits in a sentence:

The write committed, and the response was lost.

The orchestrator calls loans-api to disburse. loans-api commits. The connection resets before the 200 arrives. Did the disbursement happen? The caller cannot know. Retry blindly and you disburse twice; never retry and every network blip strands a customer mid-journey.

Under an AI agent this failure is worse, because the retry pressure is conversational — the customer says "try again," the LLM happily re-calls the tool, and now your dedup story is being exercised by a language model. This article walks the full machinery that makes that safe: an orchestration-based saga, idempotency keys done properly, a transactional outbox, and an idempotent projection — with every piece proven by chaos tests.

Reading the diagram in order:

The command — The orchestrator drives the journey and never trusts the network. It sends one command carrying one idempotency key; the system of record commits the write, the event, and the receipt in a single transaction.
Road 1 — the reply — The synchronous 200 carries the result home to durable state. This is the happy path — and the one that can vanish: the write commits, then the connection resets before the reply arrives.
Road 2 — the outbox — Because the event was written in that same transaction, it cannot not exist. An outbox relay republishes it to the event bus, and the projection applies it — a second, independent road to the exact same durable state.
Convergence — Both roads apply the same key, so duplicates collapse and lost replies replay. However many times the truth arrives, it lands exactly once. The reply can vanish; the outcome cannot.

The original article has an interactive, scroll-driven version of this diagram that lights up each step as you read.

The whole architecture is that picture: the reply can vanish, so there are two independent roads from the system of record back to durable state — and both are safe to travel twice.

Orchestration, not choreography

The loan journey — create → simulate → accept → disburse, with cancellation paths — is driven by a central command service in the orchestrator. Each step is a synchronous command against loans-api; a pure XState machine answers "is this step legal from here"; compensations (cancel the old acceptance before re-simulating; cancel simulation and acceptance when the journey is abandoned) are explicit code in the orchestrator, not emergent behavior of event subscribers.

The choreography alternative — services reacting to each other's events — was rejected for a reason that matters double under an agent: you cannot explain an emergent saga to a customer. The agent needs to answer "where are we, and what can happen next?" on every turn; a central Control Record with currentState and HATEOAS-style availableActions is that answer. One deliberately vestigial detail: disbursed is terminal and has no compensation — the cancel path for it exists only for symmetry and is never legal. Money that left is a new business process (collections), not an undo.

Idempotency: the key is the contract

Every logical command carries an idempotency key, and the scheme of those keys encodes business intent:

// Keys are deterministic PER ENTITY so a re-simulation gets a FRESH key (the old constant
// `${id}:simulation` key made loans-api dedupe a re-sim and return the stale simulation):
//   - simulation:   `${id}:simulation:${nonce}` — a fresh nonce per simulate COMMAND.
//   - acceptance:   `${id}:acceptance:${simulationId}` — one acceptance per active simulation.
//   - disbursement: `${id}:disbursement` — exactly one disbursement per journey (terminal).
export function simulationIdempotencyKey(controlRecordId: string): string {
  return `${controlRecordId}:simulation:${newId()}`;
}

Read those three lines again — they are three different answers to "what does the same command mean?" A retry of one simulate replays one nonce. A new simulate mints a new nonce (the old constant-key version deduped re-simulations into stale quotes — a real bug). And a journey gets exactly one disbursement, ever.

On the loans-api side, the envelope is brutally simple, and the simplicity is the point:

export async function runIdempotentWrite<T>(
  uow: UnitOfWork, idempotencyKey: string, responseSchema: ZodType<T>,
  produce: (tx: LoansTx) => Promise<T>,
): Promise<T> {
  return uow.transaction(async (tx) => {
    const existing = await tx.findIdempotentResponse(idempotencyKey);
    if (existing !== null && existing !== undefined) {
      return responseSchema.parse(existing);
    }
    const result = await produce(tx);
    await tx.saveIdempotentResponse(idempotencyKey, result);
    return result;
  });
}

The recorded response is saved in the same transaction as the domain write (idempotency_keys.key is the primary key — the unique constraint is the correctness proof). A replayed key returns the recorded response. There is no window where the write exists but the receipt doesn't.

Retry: bounded, classified, and invisible to the caller

With idempotency underneath, retrying becomes safe — but only worth doing for the right failures:

// Transient = a fault a RETRY could plausibly clear: a 5xx UpstreamError or a bare
// transport failure (connection reset / timeout). Terminal = any 4xx DomainError
// (validation, not-found, illegal transition): retrying only burns patience.

The schedule is five attempts, 1s base, ×2 growth — roughly 15 seconds of total backoff — with additive jitter. Additive matters: full jitter can collapse a delay toward zero, and this system's chaos test asserts elapsed > 12s under a persistent fault, proving the schedule actually ran. Jitter that can shrink the floor would make the proof flaky; jitter that only adds decorrelates concurrent retriers without touching it.

The composition is the part worth copying:

export function createRetryingLoansClient(inner: LoansClient, policy: RetryPolicy): LoansClient {
  return {
    simulate: (request) => policy.execute(() => inner.simulate(request)),
    accept: (request) => policy.execute(() => inner.accept(request)),
    disburse: (request) => policy.execute(() => inner.disburse(request)),
    /* …cancel ops… */
  };
}

A plain decorator at the composition root. The command service depends on LoansClient and does not know retries exist; the retry module knows nothing about loan semantics. One secret per module — when the schedule changes, one file changes; unit tests inject fake timers and fake randomness, production injects nothing.

The precondition is stated where it can't be missed:

// PRECONDITION: `op` MUST be idempotent. A retry may replay a call whose earlier
// attempt already committed downstream but whose response was lost (the ambiguous
// request-landed-response-lost failure). Every loans-api call satisfies this.

The transactional outbox: the event cannot not exist

Retries fix the caller's view. But the orchestrator's projection also needs the event — and publishing to a broker from inside a request handler is how events get lost. The outbox pattern closes that hole: loans-api writes the domain row and the event row in one transaction:

CREATE TABLE IF NOT EXISTS outbox (
  id TEXT PRIMARY KEY,          -- the event's UUIDv7 = dedup/order key, end to end
  event_type TEXT NOT NULL,
  control_record_id TEXT NOT NULL,
  payload JSONB NOT NULL,
  status TEXT NOT NULL DEFAULT 'PENDING',   -- PENDING -> NOTIFIED
  attempts INTEGER NOT NULL DEFAULT 0,
  published_at TIMESTAMPTZ NULL
);
-- Partial index keeps the relay poll O(pending) despite an ever-growing NOTIFIED backlog:
CREATE INDEX IF NOT EXISTS idx_outbox_pending ON outbox (id) WHERE status = 'PENDING';

One subtle move: the event id is minted first and embedded in both the HTTP response and the outbox row. The synchronous path and the async path carry the same UUIDv7, so the projection can dedup across both with a single comparison.

A DBOS-durable relay drains the table every second — claim with FOR UPDATE SKIP LOCKED, publish to SNS inside a durable step, then mark NOTIFIED guarded on status = 'PENDING'. A crash between publish and mark? The row republishes next tick. That's at-least-once by design, which is fine, because…

…the consumer is idempotent three ways

SNS fans out to an SQS queue (three failed receives → DLQ); the orchestrator's consumer applies events through the same applier the synchronous path uses:

// 1. Dedup / ordering guard. eventIds are UUIDv7 (monotonic, string-sortable): a stale or
//    already-applied event is a no-op.
if (current.lastAppliedEventId !== null && event.eventId <= current.lastAppliedEventId) {
  return current;
}
// 3. Machine-legality gate: an out-of-order backstop event … is skipped rather than
//    corrupting state.
if (!canFire(current.currentState, { type: machineEvent })) {
  return current;
}

…and the final write is a compare-and-swap on a monotonic version. Not on state — a re-simulation is a simulated → simulated self-loop, and a state-keyed CAS matches stale rows. That was a live bug; the monotonic version fixed it.

So the queue is a backstop, not a source of truth: state is durable in loans-api regardless, the synchronous path projects immediately, and nearly every queued event dedups to a no-op. The honest cost, called out in the code: the backstop carries every domain event, so it must drain full production volume just to stay at zero backlog — which is why it's competing SQS consumers rather than the original one-tick-per-second DBOS scheduled workflow that capped throughput at ~10 messages/s.

The failure, replayed end to end

Here's the full sequence, actor by actor:

Trace the scenario: simulate, attempt 1, commit, response lost.

Key CR-1:simulation:<nonce> minted once; attempt 1 commits row + outbox event E + recorded response, then the 200 dies on the wire.
The retry client classifies the transport error transient, sleeps ~1s, replays the same key.
loans-api finds the key → returns the recorded response. No second simulation, no second event.
The orchestrator projects E synchronously: version CAS 0→1, state simulated.
Meanwhile the relay publishes E → SNS → SQS; the consumer sees E ≤ lastAppliedEventId → no-op; message deleted.

Final state: exactly one simulation, one Control Record at version 1 — indistinguishable from the run where nothing failed. The e2e suite forces this exact path with Toxiproxy and asserts both the single simulation and the elapsed-time floor proving the backoff ran.

Conclusion

Nothing here is exotic — saga orchestration, idempotency keys, an outbox, an idempotent consumer are all twenty-year-old ideas. What makes the combination production-grade is that each mechanism assumes the others will be exercised: the retry assumes responses get lost, the keys assume retries happen, the outbox assumes the process dies mid-publish, the consumer assumes the queue delivers twice and out of order. And the chaos tests assume nothing — they inject the faults and measure.

If you keep one sentence: make every command mean something exact (the key scheme), make every write self-announcing (the outbox), and make every apply harmless to repeat (the idempotent projection) — then retries stop being scary and become boring. Boring is the goal.

PII That Can Actually Be Forgotten — Blind Indexes and Crypto-Shredding in an AI Agent Stack

Pablets — Tue, 28 Jul 2026 01:02:14 +0000

A DELETE doesn't delete. Postgres keeps your customer's name in heap pages until VACUUM, in every WAL segment, in every backup, in every replica. Here's an architecture where "right to erasure" is a one-key operation — and where the AI layer never gets PII to leak in the first place.

LLM systems are PII multipliers. A name typed into a chat can end up in graph state, in a checkpointer table, in logs, in traces, in a vector store — and then the customer invokes their right to erasure, and you discover that "delete the user" now means a forensic sweep across seven data stores and a backup archive you can't rewrite.

This stack takes the opposite bet, made of two architectural decisions:

1. Only one service ever holds PII. Everything else carries an opaque reference.
2. That one service stores nothing it can read without a key someone else custodies — so erasure is the destruction of one key.

The service is party-api, modeled on the BIAN Party Reference Data Directory service domain. The key custodian is OpenBao Transit ("encryption as a service" — key material never leaves it). And the AI layer gets its own, independent control: scrub before anything is stored or embedded.

Reading the diagram in order:

Everyone else — The orchestrator, loans-api, the agent, the UI — every other service holds only an opaque reference. You cannot leak what you never stored.
The vault — party-api is the one and only place PII exists — and even there it lives as ciphertext plus one-way blind indexes. It cannot read its own rows in the clear.
The keys — OpenBao Transit custodies one key per person; key material never leaves it. The vault holds the locked box; someone else holds every key.
Right to erasure — So "delete this customer" stops being a forensic sweep across seven stores. Erasure = destroy one key.
Every copy, at once — And that single deletion turns every copy unreadable at the same instant — live rows, WAL segments, base backups, replicas. Proving erasure collapses to proving one key-deletion event.
The whole bet — You can't leak what you never stored — and you can't fail to delete what a single key unlocks.

The original article has an interactive, scroll-driven version of this diagram that lights up each step as you read.

One vault everyone else references by an opaque id; one key-keeper the vault cannot live without; and erasure reduced to destroying a single key — which turns every copy, in every backup, unreadable at the same instant.

Why a row DELETE can't prove erasure

The schema's header comment states the threat model better than most compliance decks:

// Row DELETE alone cannot PROVE erasure: the plaintext lingers in Postgres pages until VACUUM, and
// in every WAL segment and base backup until those age out. So no PII column stores plaintext.
// Each PII-bearing row stores:
//
//   * `*_ciphertext`  — the row's PII as one JSON blob, encrypted under THAT PARTY's key held in
//                       OpenBao Transit. Erase destroys the key, which makes every copy of the
//                       ciphertext (live, WAL, backup, replica) undecryptable at once. Proving
//                       erasure collapses to proving one key-deletion event.
//   * `*_bidx`        — a blind index: keyed HMAC of the normalized searchable value, under a
//                       SERVICE-WIDE key that never leaves OpenBao.

That's the whole design in two bullet points: per-party keys make erasure granular; keyed hashes make search possible without plaintext. The database contains ciphertext and one-way HMACs. A dump of party-api's memory, environment, or database yields no key — Transit encrypts and decrypts on its side of the wire.

The three flows that make it work — register, lookup, erase — in full detail:

Blind indexes: equality search over data you refuse to store

"Find the party for alice@example.com" has to work — it's how a verified login binds to a bank party, and how an auditor resolves a human to a reference. The blind index is a keyed HMAC-SHA256 of the normalized value (trim().toLowerCase(), applied identically at write and at search), stored in a *_bidx column with a covering B-tree index. Search hashes the query term via Transit and does an equality match.

Two deliberate choices hide in that design:

The HMAC key is service-wide, not per-party — the code says why: "a per-party key would make 'find this email across all parties' impossible." You don't know which party you're looking for; that's the point of the lookup.
It's an HMAC, not deterministic encryption — one-way. Even if the column leaks, it reveals nothing and reverses to nothing.

And the residual risk is written down instead of wished away:

// Service-wide HMAC key for blind indexes. NEVER destroyed by erasure: erasing a party deletes its
// blind-index ROWS instead. Residual risk, accepted and documented: a blind-index hash surviving in
// an aged DB backup is pseudonymous (keyed hash, key held only by OpenBao) but not erased until
// that backup expires — backup retention is the governing control there.

Erasure: key first, rows second — because of how each half fails

The erasure endpoint is the one irreversible route in party-api, and it executes in a precise order:

async erase(partyReference) {
  await requireLive(deps.db, partyReference);
  await deps.cipher.destroyKey(partyReference);        // 1. crypto-shred
  const erasedAt = deps.now();
  return deps.db.transaction(async (trx) => {          // 2. tombstone
    // Whole rows, not nulled columns — a column added later cannot be forgotten by a DELETE.
    for (const table of PII_TABLES) {
      await trx(table).where({ party_reference: partyReference }).del();
    }
    await trx(TABLES.ENTRY).where({ party_reference: partyReference }).update({
      status: 'Erased', erased_at: erasedAt, /* … */ person_details_ciphertext: null,
    });
  });
}

The ordering is a failure-mode argument, not a style choice: destroy the key first, and a crash before the row-delete leaves a state that a retry completes. Delete rows first, and a crash before key destruction could strand a permanently half-erased party — whose retry the requireLive guard then refuses. When an operation is irreversible, you design it around where it can be interrupted.

What remains is a tombstone: the entry survives with status: 'Erased', an erasedAt date, and no identity whatsoever. That's deliberate — every other service still holds the partyReference, and a dangling reference must be explainable ("erased on 2026-07-17"), not indistinguishable from data loss. Audit history everywhere keeps every structural fact — party, amount, state, time — and permanently loses the ability to say who.

Downstream, the orchestrator's party gate closes the loop: a new journey for an erased party is refused with a 422, fail-closed, while historical journeys keep pointing at the reference by design.

Who may push the button

Erasure is authenticated with the same rigor as the data it destroys. The route demands a bearer token, verified against Keycloak (RS256, pinned issuer, email_verified, authorized-party check), and then applies a two-ground rule:

if (principal.realmRoles.includes(complianceRole)) {
  return;                                   // ground 1: compliance-officer
}
const own = await config.resolver.referencesForEmail(principal.email);
if (own.includes(targetPartyReference)) {
  return;                                   // ground 2: it's your own party
}
throw new ForbiddenError(
  'This identity may erase only its own party; erasing another party requires a compliance role.',
);

No token → 401. Unverifiable token → 401 (the reason stays in the logs; the wire message is deliberately coarse). Valid token, wrong person → 403. The e2e suite curls all three outcomes and then asserts the tombstone directly in the database: entry present, status Erased, zero PII rows.

Note the elegant self-reference: ground 2 uses the blind index itself — the verified email is hashed and matched to prove "this is your own party." The privacy mechanism doubles as the authorization mechanism.

The AI plane: don't shred what you never stored

Crypto-shredding protects the system of record. The conversational plane — LLM prompts, LangGraph checkpoints, the self-improving RAG — is a different data plane with its own control, applied at a single ingestion seam:

def _redact_and_log(text: str) -> str:
    """
    Scrub PII from inbound USER text at the single ingestion seam — before it becomes
    a message in graph state (which the checkpointer persists) or reaches the LLM.
    Logs the categories redacted, never the values.
    """

The redactor is pure regex + Luhn: SSNs, payment cards (Luhn-validated so a $20,000 loan amount is never mistaken for a card number), keyword-anchored account numbers, passwords and PINs. Before anything reaches the pgvector knowledge base, a second pass generalizes the text — every numeric collapses to <num> — so the KB stores problems and solutions, never people or amounts. And Bedrock Guardrails add a model-boundary backstop, anonymizing emails and phones and blocking cards and SSNs in both directions.

The two planes compose into a tidy claim: the agent never sends raw identifiers to a model or a vector store, and the system of record never stores plaintext identity at all. Erasure therefore touches exactly one service, and proving it means producing one key-deletion event.

Conclusion

Most "GDPR-ready" architectures are a promise to go looking for data when asked. This one is a refusal to spread it: one PII service, opaque references everywhere else, ciphertext under per-party keys, search via one-way hashes, erasure as key destruction plus a tombstone, and an AI layer that scrubs before it stores.

The takeaway generalizes past banking: decide where PII is allowed to exist before you build, make that place unable to read its own database without external keys — and then the right to be forgotten stops being an archaeology project and becomes an API call.

Guardrails for AI Agents — Five Deterministic Rings Between an LLM and Real Money

Pablets — Tue, 28 Jul 2026 01:01:41 +0000

Prompts are suggestions. Guardrails are architecture. How a loan-acquisition agent layers a deterministic flow, an MCP contract, ownership gates, a pure state machine, and idempotent writes so that the LLM can be wrong safely.

Every "agent gone rogue" postmortem has the same shape: the LLM did something surprising, and there was nothing between the surprise and the consequence. The prompt said "only disburse after the customer accepts" — and the prompt lost.

The fix is not a better prompt. It's accepting a design rule:

A guardrail is only a guardrail if the LLM cannot route around it. Anything the model can skip by phrasing a tool call differently is a convention, not a control.

This article walks the guardrail stack of a production-grade conversational loan agent — LangGraph agent, MCP tool layer, Node orchestrator, banking core — from the outside in. The whole system runs offline with docker compose up, and the guardrails are proven by end-to-end tests that attack them.

Reading the diagram in order:

The proposal — The LLM proposes a command — «disburse the loan». From here it has no more power: the proposal must survive five gates, each owned by a different module with a different reason to change.
Gate 1 — conversation — Are the slots valid and the terms accepted? If not, the answer is "keep talking" — the flow stays in intake and never advances on a terse follow-up the router misread.
Gate 2 — identity — Is this a proven caller acting on their own party? No verifiable token, no journey: the gate returns 401 and nothing is written.
Gate 3 — ownership & rules — Is it your loan, is the party still alive, does policy pass? A "no" here is a 403, 404, or 422 — a machine-readable refusal, not an exception.
Gate 4 — state machine — Is this a legal step right now? A pure XState machine answers canFire(state, DISBURSE). Out of order gets a 409 — the model cannot talk its way past the lifecycle.
Gate 5 — idempotency — Is this the first time this exact command has run? A duplicate replays the first answer instead of moving money twice. Ask twice, pay once.
Money moves — Five yeses, and only then does the money move — exactly once. One straight road to the money, with five places it can be stopped and only one where it can complete.
Every "no" is data — And every refusal returns to the agent as a code plus a reason it explains in the chat — never an exception, never a write. The LLM can be wrong, safely.

The original article has an interactive, scroll-driven version of this diagram that lights up each step as you read.

One proposal, five yes/no gates, one irreversible action. A "yes" moves the command one step closer to the money; every "no" comes back as data — a code plus a reason the agent explains in the chat, never an exception, never a write. Each gate is owned by a different module with a different reason to change. Let's walk them outside-in.

Ring 1 — the agent itself is deterministic where it matters

The LangGraph agent does not run a free ReAct loop over the loan tools. The loan lifecycle is a deterministic, turn-resumed intake node; the LLM is quarantined to two narrow jobs — proposing an intent classification, and phrasing help text. Neither can advance the flow.

The most instructive piece is the sticky intake routing guardrail:

"""
Route based on state + the classified intent from the router node.

- intake_active (sticky): once we're collecting a loan, STAY on the intake path
  regardless of how the router classifies a terse follow-up answer (a bare "24").
  This is the flow's "State Enforcement — the orchestrator overrides the LLM".
"""
if state.get("intake_active"):
    logger.debug("Routing to intake (intake_active sticky guardrail)")
    return "intake"

Without this, a customer answering "24" to "how many months?" gets misclassified as small talk and the flow derails. With it, the router's opinion is advisory once a journey is underway.

The intake node also pre-validates slots against real policy before any tool call — and here's the decomposition detail worth stealing: the agent and the orchestrator call the same shared policy function, so the conversational check and the authoritative check can never drift apart:

// Pure policy check, shared so the agent and the orchestrator apply IDENTICAL rules. Returns the
// list of human-readable violations (empty = admissible), so the orchestrator can put them in a
// BUSINESS_RULE_VIOLATION detail and the agent can echo them conversationally.
export function checkAgainstRules(input, rules: LoanRules): string[] {
  const violations: string[] = [];
  if (input.amountMinor < rules.minAmountMinor) { /* … */ }
  if (input.amountMinor > rules.maxAmountMinor) { /* … */ }
  if (!rules.allowedInstallments.includes(input.termMonths)) { /* … */ }

Finally, the agent shows personalized disclosures and an explicit "I accept / Cancel" gate before acceptance — deterministic template renders of the customer's exact numbers, deliberately not retrieval. A disclosure that an LLM paraphrases is a disclosure a lawyer can't sign off on.

Ring 2 — the MCP layer: a contract designed for an unreliable caller

The FastMCP server (loan-mcp) is where agent-facing design lives: docstrings carry ordering hints ("Call this FIRST", "Legal only after accept_terms"), and — crucially — errors are data, not exceptions:

"""
Tools pass through the orchestrator's value object on success. On failure we
return {"error": <RFC 9457 problem>} instead of raising so the agent receives
a usable message — the whole problem (type/title/status/detail/code/instance/
traceId) rides along, and a 409 STATE_TRANSITION reads as an out-of-order call
the guardrail refused, not an opaque exception.
"""
except OrchestratorError as exc:
    if exc.is_state_transition:
        logger.warning(f"[loan-mcp] {action_label} refused by guardrail: {exc.message}")
    return {"error": exc.as_dict()}

A refusal that reaches the model as structured data becomes a conversational event: "we can't disburse yet — you haven't accepted the terms." A refusal that dies as an exception becomes a stalled chat.

This layer is also the identity seam: with LOAN_MCP_AUTH_ENFORCED=true, no verifiable token means no journey — and a valid token binds the journey to the caller's own party, resolved from the verified email. Fail closed:

"""
* No token + enforcement ON  -> refused (401): the composed stack fails closed, so a
  journey can never open against an unproven identity.
* A token -> verified against Keycloak's JWKS (signature/issuer/expiry/email_verified),
  then bound to the CALLER'S OWN party.
"""

Ring 3 — the orchestrator's gates: ownership, party, rules

Every command that reaches the orchestrator's REST edge passes three gates in order, before any state machine question is even asked:

Ownership — the forwarded token is independently re-verified (a second, deliberately separate implementation from loan-mcp's), and the token-derived party must match the customerRef being operated on. Errors are coarse on purpose: specifics go to logs, an attacker learns nothing.
Party gate — the party must exist and be live. An erased (crypto-shredded) party yields a 422; unknown yields 404. If party-api is unreachable, the gate returns 503 — fail closed, never "assume it's fine."
Rules gate — the authoritative policy check (same shared function as Ring 1), returning 422 with the human-readable violations.

The trade-off is explicit and named: ORCH_AUTH_MODE runs off / verify-when-present / require. The compose stack uses verify-when-present so token-less e2e and internal tools keep working; a real deployment runs require. The point is that the posture is a configuration decision, not an accident of middleware.

Note the asymmetry with Ring 1: identity and policy dependencies fail closed; conversational helpers fail open. The agent's draft check degrades gracefully because "the orchestrator still enforces authoritatively on create" — but the orchestrator's gates never degrade, because there is nothing behind them.

Ring 4 — the pure state machine: the only authority on "may this happen"

The heart of the system is an XState machine that is pure to the point of asceticism:

// The loan-acquisition lifecycle as a PURE, deterministic XState machine. It is the
// guardrail: given the current durable state and a candidate event (proposed by the
// non-deterministic LLM, or projected from a loans-api domain event), it answers "is this
// transition legal?" and "what is the next state?". It performs NO side effects — the
// synchronous command handlers and the SQS backstop execute the actual step and then
// project this machine only on success. Coupling points one way: everything depends on the
// machine; the machine depends on nothing. SECRET: the legal state/event transition graph.

The disburse handler shows what "layered refusal" looks like in practice — each check has its own status code, so the caller learns exactly which ring refused:

if (!canFire(current.currentState, { type: 'DISBURSE' })) {
  return fail('STATE_TRANSITION', `cannot disburse from state ${current.currentState}`, 409);
}
if (current.snapshot === null) {
  return fail('BUSINESS_RULE_VIOLATION', 'no simulation snapshot to disburse against', 422);
}
if (isSnapshotExpired(current.snapshot, nowIso())) {
  return fail('BUSINESS_RULE_VIOLATION', 'the simulation snapshot has expired; re-simulate before disbursing', 422);
}

Because the machine is pure and depends on nothing, the same legality function serves three callers: the synchronous command handlers, the SQS backstop's projection applier (so an out-of-order replayed event is skipped, never applied), and a non-mutating /signal check endpoint that lets you measure classifier→guardrail precision without touching state.

Ring 5 — loans-api: even a legal command must be safe to repeat

The innermost ring assumes everything above it failed. The system of record wraps every write in an idempotency envelope (same transaction as the domain change), and adds a relational belt-and-suspenders check — the acceptance being disbursed must actually belong to the active simulation. A retried or duplicated command replays the recorded response; it never writes twice.

Underneath all five rings sits durability: the Control Record is persisted via DBOS on Postgres, advanced only through a version compare-and-swap plus per-event dedup. The version-CAS detail is a war story in itself: the CAS used to key on state, and a re-simulation is a self-loop — simulated → simulated — so the CAS silently matched stale rows. Monotonic version fixed it. If your state machine has self-loops, a state-keyed CAS is a bug you haven't met yet.

The attack, end to end

Here's the full detail of the five rings — who checks what, in which order, and what the happy and rejected paths look like:

What happens when something — a creative LLM, a curious tester, a direct REST call — tries disburse while the journey is still simulated?

canFire('simulated', DISBURSE) → false. Nothing mutates; loans-api is never called.
The orchestrator returns RFC 9457 application/problem+json, status 409, code STATE_TRANSITION.
loan-mcp passes it through as a value: {"error": {code, status, title, …}}.
The agent reads the code and explains conversationally — "let's review the terms first."
Durable state remains simulated; acceptanceId and disbursementId remain null.

Every step of that path is asserted by e2e tests, over both REST and MCP — including that the MCP tool call resolves (rather than throws) and that the durable record is untouched. A guardrail without a test that attacks it is a hope.

The soft outer filter, and why it's allowed to be soft

Outside the five rings sits one more layer: a Bedrock Guardrails config for model-level content filtering. Its deny-topics are domain-specific — unlicensed financial advice ("should I put this loan into crypto?") and lending fraud and evasion ("how can I inflate my income to qualify?") — alongside harmful-content filters and PII masking. The customer-facing refusal copy lives in a single module-level constant, so the guardrail definition sent to Bedrock and the local fallback message can never drift apart.

Unlike everything inside it, this filter fails open on error — acceptable only because it is not, and must never be, the authoritative control. One operational note: guardrails are created-or-fetched by name, so a policy change in the repo reaches an existing environment only when the guardrail is recreated or a new version is published.

That's the real lesson of the ring model: you can afford a soft, imperfect outer filter precisely because the inner rings are hard. Put your rigor where the irreversibility is.

Conclusion

The five rings are not redundancy for redundancy's sake — each one hides a different secret and refuses a different class of mistake:

Ring	Refuses	Mechanism
Agent flow	conversational derailment	sticky routing, shared pre-validation, explicit confirm
MCP contract	unusable failures, unproven identity	errors-as-data, fail-closed party binding
Orchestrator gates	wrong caller, dead party, bad policy	ownership + party + rules, fail closed
XState machine	illegal ordering	pure `canFire`, 409 as data
loans-api	duplicated execution	idempotency envelope + relational check

The LLM sits outside all of them, free to be brilliant and wrong. You don't make an agent safe by prompting it better — you make it safe by building rings it cannot route around.

AGUI with Chainlit — Building a Production Chat Frontend for an AI Agent

Pablets — Tue, 28 Jul 2026 00:58:22 +0000

The agent GUI is where every backend shortcut becomes visible: tokens expire mid-chat, resumed threads render blank, streamed answers appear twice. Here's a Chainlit frontend that survives production — real OIDC with a two-hostname split, mid-chat token refresh, persisted threads with interactive Plotly charts, and the fixes for the failure modes you only meet after you persist.

Agent demos treat the UI as an afterthought — a chat box taped over an API. Then real users arrive, and the UI turns out to be the layer with the most state: login sessions, websocket connections, streamed partial messages, thread history, rich widgets that must survive a page reload three weeks later.

This article walks the Chainlit frontend of a conversational loan agent. The design constraint that shapes everything: the UI is domain-agnostic. It POSTs {prompt, customer_name, conversation_id} to the agent's /invocations endpoint and renders the streamed response — the same code path serves a local container and a deployed AWS Bedrock AgentCore runtime, switched by one env var. All loan logic lives behind that endpoint; the UI's job is transport, identity, rendering, and persistence.

Reading the diagram in order:

One dumb pipe — The UI is domain-agnostic: it POSTs {prompt, customer_name, conversation_id} to the agent’s /invocations endpoint and renders the streamed events. All loan logic lives behind that pipe — the same code serves a local container and a deployed AgentCore runtime.
Identity — A real authorization-code login against Keycloak — no demo password gate — with tokens that renew mid-chat before they expire.
Streaming — Answers render as they are thought: streamed SSE chunks, not a spinner and then a wall of text.
Memory — Threads resume weeks later — charts and all. Persisted sessions, steps, and elements that survive a page reload three weeks on.
Rich answers — Interactive Plotly charts, choices, and cards — the amortization schedule is a widget, not a paragraph.
The one rule — The UI owns the experience, never the domain. Everything hard about it is state: sessions, streams, storage, resume.

The original article has an interactive, scroll-driven version of this diagram that lights up each step as you read.

Four jobs — identity, streaming, memory, rich answers — around one conversation, over one dumb pipe. The rest of this article is those four jobs, in the order the failure modes taught them.

Login: one identity provider, two hostnames

The UI runs a real authorization-code flow against Keycloak — no demo password gate. The classic compose-stack trap is the two-hostname split: the browser reaches Keycloak at https://localhost:8543, while the UI's backend must exchange the code in-network at https://keycloak:8443. Chainlit's stock Keycloak provider derives every URL from a single base — it literally cannot express this. So the app registers a small subclass:

def __init__(self):
    # base_url is what the PARENT uses for the token + userinfo exchanges (in-network).
    self.base_url = (os.environ.get("OAUTH_KEYCLOAK_INTERNAL_URL") or "").rstrip("/")
    # authorize_url is what the BROWSER is redirected to (host-published).
    public_url = (os.environ.get("OAUTH_KEYCLOAK_PUBLIC_URL") or "").rstrip("/")
    self.authorize_url = (
        f"{public_url}/realms/{self.realm}/protocol/openid-connect/auth"
    )

Two more login details earn their place through failure:

prompt=login on every authorize. Chainlit's /logout clears only its own cookie; Keycloak's SSO cookie survives, so "log out" followed by "Continue with Keycloak" would silently wave the user back in — wrong for a banking flow. Forcing a fresh credential prompt also kills an instant-SSO redirect that raced Chainlit's oauth_state cookie check.
The UI serves HTTPS for the same reason: when it ran plain HTTP next to an HTTPS Keycloak, the oauth_state cookie went cross-scheme under schemeful same-site rules and login failed intermittently — the worst kind of bug.

Token refresh: a chat outlives its access token

A loan conversation takes longer than a 30-minute access token. Rather than failing closed mid-chat (which this stack's downstream will do — every hop verifies the forwarded token), the UI renews in place before each turn when the token is expired or within a 60-second skew window:

new_access = token_response.get("access_token")
if not new_access:
    raise _ReloginRequired()
metadata["access_token"] = new_access
# Keycloak issues a rotated refresh token on each grant — keep the newest so the next
# renewal uses a valid one (matters when the realm revokes reused refresh tokens).
if token_response.get("refresh_token"):
    metadata["refresh_token"] = token_response["refresh_token"]

If the refresh fails — expired, revoked, network — the turn handler removes the empty placeholder bubble and shows a transient "please sign in again" toast instead of persisting a dead error line into the thread or forwarding a token that would 401 three services later. Failure UX is part of the auth design.

(Trade-off, named in the code: access and refresh tokens live in the Chainlit user's metadata JSONB — a token at rest, acceptable for a demo, replaced by a server-side session store in production.)

The turn: one SSE stream, many event types

The full picture of a turn — streaming, persistence, and the resume path we'll meet below:

Each turn POSTs to /invocations and reads Server-Sent Events. The dispatch loop switches on a small event vocabulary — chunk, choices, simulation_snapshot, disbursement_details, blocked, error — and maps each to a UI affordance: streamed tokens into the live bubble, radio pickers via AskActionMessage, markdown cards, and the crown jewel:

await cl.Message(
    content=("Here's the full breakdown. This chart is **exactly what you'll pay** …"),
    elements=[cl.Plotly(name="amortization", figure=figure, display="inline")],
).send()

An interactive amortization chart — principal/interest split per month, balance line, a range slider — as a first-class chat element. This is the argument for a chat framework over a chat box: cl.Plotly, action pickers, and thread history come free, and the app code stays domain logic.

One streaming war story lives on the agent side but surfaces here: the flow node composes internal LLM helper calls and then appends the final message itself, so every helper answer streamed twice — once as tokens, once as the appended message. The fix is a tagging pattern (tags=["loan_flow_internal"]) with the streaming layer dropping tagged spans. If you compose LLM calls inside LangGraph nodes and stream messages mode, you will meet this bug.

Persistence: the part everyone underestimates

Chainlit's SQLAlchemy data layer persists users, threads, steps, elements, and feedback — into the stack's dedicated chainlit-db Postgres instance (database-per-service applies to the UI too; chat history and business data have different owners, lifecycles, and erasure stories). But the data layer creates nothing itself, and this is where two production bugs earned their scars.

Bug 1: steps silently stopped persisting. The app bootstraps the schema with CREATE TABLE IF NOT EXISTS — which never adds columns to an existing table. A database first created under an older Chainlit lacked the autoCollapse column that a newer Chainlit's create_step INSERTs; Postgres raised UndefinedColumnError, and steps — the chat history itself — quietly failed to save. No crash. Just missing history. The fix is an idempotent migration pass at startup:

# Idempotent migrations for tables that already exist from an older schema. CREATE TABLE IF NOT
# EXISTS never adds columns, so newer Chainlit columns (e.g. autoCollapse, added in 2.11) must be
# back-filled here — otherwise create_step fails with UndefinedColumnError and steps (including the
# per-callback run steps) silently fail to persist.
_CHAINLIT_MIGRATION_DDL = (
    'ALTER TABLE steps ADD COLUMN IF NOT EXISTS "autoCollapse" BOOLEAN',
)

Bug 2: resumed charts rendered blank. Live, the Plotly figure renders perfectly. Reopen the thread a day later: an empty rectangle. Root cause: Chainlit stores element blobs under extension-less keys, and its /public file route guesses Content-Type from the filename — no extension, application/octet-stream, and the frontend's Plotly component renders nothing. The fix — in the app's custom filesystem storage client — appends the mime's extension to the key at upload time, so the stored key, the on-disk file, and the served Content-Type stay consistent:

@staticmethod
def _with_ext(object_key: str, mime: str) -> str:
    """Append the mime's file extension so /public serves the right Content-Type."""
    ext = _MIME_EXT.get(mime) or mimetypes.guess_extension(mime or "") or ""
    return object_key if not ext or object_key.endswith(ext) else object_key + ext

The general lesson from both: persistence bugs are resume bugs. Everything works live, because live rendering never touches the storage path. You only find out weeks later, from a user, unless you test the reload.

Two smaller persistence notes worth stealing: the app adds its own FK indexes (Chainlit ships none beyond primary keys — a thread resume would seq-scan children), and it detaches Chainlit's per-callback wrapper step so assistant messages persist top-level instead of orphaned under a never-persisted parent.

Resume: one id ties the whole stack together

cl.context.session.thread_id — Chainlit's own thread uuid — is reused verbatim as the agent's conversation_id. On resume, the UI rebinds that single id, Chainlit replays the persisted messages and elements, and the agent continues from the same durable LangGraph checkpoint. UI history and agent memory can't drift apart, because they share one identity. (Bonus: at 36 characters it also satisfies AgentCore's runtimeSessionId ≥ 33 requirement in cloud mode.)

[SCREENSHOT: a resumed thread showing chat history and the interactive amortization chart rendering correctly]

Conclusion

None of this is glamorous, and that's the point. The chat frontend is a distributed-systems participant like everything else: it holds tokens that expire, writes to a database that drifts, serves blobs whose content types matter, and re-enters conversations that must resume exactly. Chainlit gives you the chrome — websockets, streaming, OAuth plumbing, thread history, rich elements — and leaves you the responsibilities: owning your schema migrations, your token lifecycle, your storage semantics, and your resume path.

Treat the AGUI as part of the architecture, test the reload and the re-login as seriously as the happy turn, and the UI stops being where your backend's rigor goes to die.

Teach While Selling — Agentic RAG and the Death of the FAQ Page

Pablets — Tue, 28 Jul 2026 00:58:20 +0000

For twenty years, education and conversion lived in different places: the landing page persuaded, the FAQ explained, the docs detailed, the legal PDF disclosed — and the application form converted, alone. A conversational agent collapses all of them into one channel. This article is about the "how": answering a customer's fears at the exact moment they appear, inside the flow, without derailing it — and the two-memory, human-reviewed architecture that keeps those answers trustworthy.

Watch a customer abandon a loan application and you'll almost never see them fail at the form. They fail at a question the form can't answer: "wait — what does French amortization actually mean for what I pay?", "is 24 months better for me than 36?", "what happens if I want to pay early?"

The old web's answer was spatial: put the explanation somewhere else. A Q&A page. A glossary. A "Learn about loans" hub. A 40-page terms PDF. Every one of those is a context switch, and every context switch is an exit — the customer leaves the flow to resolve their uncertainty, and a meaningful fraction never comes back. The conversion funnel and the education content were two different products, built by two different teams, meeting only in analytics dashboards.

A conversational agent dissolves that separation, and this is genuinely new:

The same turn that collects the loan amount can teach what an amortization schedule is — personalized to this customer's product, numbers, and moment of doubt — and then continue the sale exactly where it paused.

Selling and teaching stop being two flows. The guided-selling literature has been circling this for years — consultative bots that "educate customers on why a product fits" are the digitization of the good in-store salesperson — and it matters most precisely in high-consideration purchases like credit, where research friction kills conversion. But doing it safely in a regulated domain raises the question this article actually answers: when the agent teaches, where do its answers come from, and why should anyone trust them?

Reading the diagram in order:

A customer gets stuck — Mid-flow, a slot parse fails — «what’s the max I can borrow?» The value the flow needs isn’t in the answer.
Layered help answers — Help is tried in order of trust: 📗 authored knowledge first, then 💬 learned memory, then a sandboxed generated fallback. The cheapest, safest layer that can answer, wins.
The flow moves on — The answer resolves and the flow advances on a valid value. Help shapes the words — it never touches the state machine.
Scrubbed & templated — Only then is the resolution captured — and never raw. It is scrubbed and templated first: no names, no numbers, just {placeholders}.
A human signs it — And a UX writer approves it. Nothing unreviewed is ever served — the learning is free, but the safety is designed in.
The next customer — Then the next customer to get stuck the same way gets that answer — rendered with today’s numbers, not the ones it was learned from.
The loop closes — And they’re back at the start, one step smoother. Every resolved conversation makes the next one easier — the loop that teaches itself.
The one invariant — The agent’s own conversations become its knowledge — but help shapes the words, never the state, and a human signs every reused answer.

The original article has an interactive, scroll-driven version of this diagram that lights up each step as you read.

That's the whole idea in one loop: a stuck customer, a layered answer, a resolution — scrubbed of everything personal, templated so no number is frozen in time, signed off by a human UX writer, and then served to the next person who hits the same wall, rendered with today's product values.

The "how", part 1 — detect the teaching moment at the point of friction

The channel-merge is only valuable if the agent notices when to switch modes. This system's loan flow is a deterministic slot-filler (amount, term, product) — and every failed parse of a customer reply is routed through an intent classifier first:

"50000" → a value. Fill the slot, move on. Selling.
"what's the maximum I can ask for?" → a question about the slot. Teaching moment.
"how does French amortization work?" → a conceptual question. Deeper teaching moment.

This is the capillarity the channel gives you: the old web could only educate at page granularity — a FAQ entry someone must find. The agent educates at slot granularity: it knows the customer is stuck on term_months, for this product, having already said this amount. The teaching moment arrives pre-contextualized, at the narrowest possible point of friction. No FAQ page can know where you stopped typing.

The "how", part 2 — teaching must never derail the sale

Here's the design rule that makes mixing the flows safe rather than chaotic:

The help resolver owns no transition logic — it returns only the help TEXT. The state machine still advances only on a validly-parsed, in-policy value.

Answering "what's the maximum?" does not fill the slot, skip a step, or advance the journey. The flow holds its place; the answer arrives; the question is re-asked with the customer's context intact. Education is a detour that isn't a detour — the sale is exactly where it was, one uncertainty lighter. This is what lets the two previously-separate flows coexist in one channel without corrupting each other: the teaching layer is architecturally incapable of touching the selling layer's state. (And below both sits the orchestrator's XState guardrail, which would refuse an illegal transition anyway.)

The second half of the rule is customization: the teaching is rendered from the customer's actual situation. Deterministic per-slot guidance is built from the real product's min/max/installments; the LLM helper gets those constraints injected as facts it may rephrase but not invent; disclosure summaries are deterministic template renders of the customer's exact numbers — deliberately not retrieval, because a paraphrased disclosure is one a lawyer can't sign. A FAQ page speaks to everyone; this speaks to you.

The "how", part 3 — two memories, because trust is not one thing

Now the question you should ask of any system that "answers customer questions while selling credit": says who?

This stack's answer is architectural: two physically separate knowledge stores with different trust contracts, different write paths, and different reasons to change — plus a strict precedence between them, and a human review gate on everything the system taught itself. The full machinery — the cascade, the two collections, and the write → review → read learning loop:

Store 1 — knowledge_docs: the curated corpus. Authored explainers (French amortization, repayment basics, terms and conditions), chunked by heading, ingested idempotently by content hash. This is assumed-verified content: written, refined, reviewed — the same editorial standard as the old docs site, now retrievable mid-conversation. It changes only when a human changes a document.

Store 2 — slot_help_kb: the learned memory. Question→solution pairs harvested from the agent's own resolved conversations — the self-information injection loop. It changes every time a customer gets unstuck. Cheap, self-scaling, and unreviewed at capture — which is exactly why a freshly-learned answer is a candidate, and candidates are never served. Between capture and reuse sits a human.

The cascade encodes the trust ordering — curated answers first, approved learned memory second, fresh generation last — and the write path always lands in the review queue:

explanation = await _knowledge_answer(slot, text)       # 1. curated corpus first → 📗 marker
if explanation:
    return explanation

key = generalize(text)                                  # 2. APPROVED learned memory second
hit = await rag_store.retrieve(key, slot)               #    (status='approved' only)
if hit and hit.get("solution"):
    # Stored solutions are TEMPLATES ({min_amount}, {terms}, ...) so no number is ever frozen
    # at capture time; render fills the CURRENT product's values. A template this context
    # cannot fill (render -> None) is treated as a miss — generate fresh, don't serve holes.
    rendered = help_templates.render(str(hit["solution"]), product=product, products=products)
    if rendered:
        return f"{_MARK_LEARNED}\n\n{rendered}"          #    → 💬 marker

solution, source = await _generate(slot, text, messages, product, products)
# Stored PARAMETERIZED (concrete values -> placeholders) as a status='candidate' row: it enters
# the UX-writer review queue and is never served until approved.
if hit is None:
    template = help_templates.parameterize(solution, product=product, products=products)
    await rag_store.remember(key, template, slot, source)  # 3. generate — then queue for review
return solution

And crucially, the learned store never writes into the curated one. The code states the boundary as a decomposition principle: the two collections have "a different reason to change: authored knowledge vs learned conversation memory," sharing only the embeddings module and the Postgres server.

Is the two-store split actually right? What the outside world says

I went looking for external evidence, and the pattern holds up from three independent directions:

The support industry already formalized it. The KCS (Knowledge-Centered Service) methodology — the dominant knowledge-management practice in support organizations — captures knowledge from live interactions as draft/candidate articles, formally distinct from validated ones, and treats "confidence visible to the user" as a core requirement. Constraining AI retrieval to validated content is explicitly cited as a hallucination control. Our knowledge_docs vs slot_help_kb is precisely KCS's validated-vs-candidate split, expressed as two vector collections.
Commercial AI-support platforms enforce the same tiering. Intercom's Fin, for instance, answers only from explicitly approved content sources, with separately-managed curated "snippets" — source control is the product's trust story.
The research literature warns about exactly the failure the split prevents. Work on RAG "self-citation" and knowledge-base poisoning shows that indexing an assistant's own outputs into the authoritative corpus creates feedback loops — "generate-evaluate-reinject" cycles that amplify bias and quietly degrade truth. The mitigation the literature converges on is the one this architecture hard-codes: keep generated content out of the authoritative corpus, stamp provenance, and gate reuse.

So yes — the two-store design isn't a quirk; it's the intersection of a support-industry methodology, commercial best practice, and the published failure modes of self-learning RAG.

What the implementation does to earn that trust

Checking the code against that external bar, the trust seams that matter are present:

Separation and precedence — learned content can never shadow curated content, because the curated corpus is consulted first and the stores never mix.
A draft→validated lifecycle, with humans in it. Every learned answer lands as status='candidate' — and candidates are never served. A dedicated curation dashboard (its own service, Keycloak-gated on a ux-writer realm role, fail closed) is where UX writers approve, edit, or reject; the agent retrieves only status='approved' rows, and every decision records who reviewed it and when. This is exactly the KCS candidate/validated split, with the dashboard as the "content health" loop.
Templates, not frozen numbers. Before storage, concrete values are parameterized into placeholders — You can borrow between {min_amount} and {max_amount} — and re-rendered with the current product's values at serve time. Change the product's limits tomorrow and yesterday's approved answer says the new numbers. A template the current context can't fill isn't served with holes; the system falls back to generating fresh. (This extends the disclosures' template discipline to the learned tier — dynamic data always comes from live context, never from capture time.)
Provenance the customer can see. Curated-corpus answers are marked "📗 From our loan guides"; approved learned answers "💬 A reviewed answer to a similar question"; fresh generation carries no marker. The trust tiering reaches the user, not just the architecture — KCS's "make confidence visible," in one line of markdown.
Provenance on every learned row — a source column (llm vs deterministic) plus an embedding signature (provider, model, dim) that every query filters on, so a provider switch starts a fresh logical KB instead of silently comparing incompatible vector spaces. Reviewing edits text and status only — an approval never moves a row in vector space.
Sanitization before storage — every learned key passes pii.redact (SSNs, Luhn-validated cards, accounts, passwords, PINs) and then collapses all numerics to <num>, so the KB stores shapes of problems, never people or amounts. "Can I borrow <num> for <num>" is one lesson regardless of whose money it was.
Confidence gating — reuse requires cosine similarity above a provider-aware threshold (auto: 0.35 for the offline hash embeddings, 0.83 for real models); below it, the system generates fresh rather than serving a weak match.
Bounded blast radius — the worst a bad learned answer can do is be unhelpful text. It cannot move the state machine, and everything fails soft to the deterministic guidance layer if pgvector is down.

One decomposition note on the dashboard, because it's easy to get wrong: it shares no code with the agent — the slot_help_kb table and the placeholder vocabulary are the entire contract. Deliberately so: the agent's store is fail-soft (a broken KB must never break a customer turn), while a review tool must fail loud (an Approve that silently does nothing is worse than an error page). Opposite failure semantics are a different reason to change, hence a different module.

The full trust gradient

Zooming out, the teaching channel actually runs on four tiers, strictest first — and the tier is chosen by what the content is, not by what's convenient:

Tier	Content	Source of truth	May the LLM rephrase it?
Disclosures & terms	legal text, the customer's exact numbers	deterministic templates	No — rendered verbatim
Concepts	"how does amortization work?"	curated `knowledge_docs`	Synthesize from chunks (real LLM) or serve verbatim (offline)
Recurring confusion	"what's the max?" phrased a thousand ways	learned `slot_help_kb`	Only UX-writer-approved templates, re-rendered with live values; threshold-gated
Novel questions	everything else	sandboxed generation + deterministic fallback	Yes — words only, constraints injected, never a value

That table is the answer to "how do you teach while selling in a regulated domain": match each kind of teaching to the strictest source that can produce it, and let none of them touch the sale's state.

Conclusion

The channel-merge is the prize: one conversation that persuades, explains, discloses, and converts — with the explanation arriving at slot-level granularity, personalized to the customer's own numbers, at the exact moment doubt appears. That's something no arrangement of landing pages, FAQs, and PDFs could ever do, and it's why this communication channel is worth the engineering.

But the merge is only safe because the knowledge behind it is tiered: curated docs you stand behind, learned lessons you scrub, template, and put a human signature on before anyone sees them, generation you sandbox — and legal text you never let a model touch. Keep the two memories separate, keep the precedence strict, keep help powerless over state, and put a UX writer between what the agent learns and what it repeats — then the same chat can be your best salesperson and your best teacher at once, without ever being your compliance department's worst day.

Sources (two-store validation and channel thesis):
KCS Article State — Consortium for Service Innovation · KCS in the Age of AI · What is Knowledge-Centered Service · When RAG Starts Citing Itself · Bias Amplification in RAG: Poisoning Knowledge Retrieval (arXiv) · Curated retrieval vs open web search: a coverage–trust trade-off (arXiv) · Intercom Fin — content sources · Intercom Fin — snippets · Guided Selling in the Age of Conversational Commerce (commercetools) · AI Chatbots in E-Commerce: From FAQ Bot to Digital Sales Consultant

Build a Production-Grade Conversational Loan Agent — LangGraph, MCP, and Durable Banking Orchestration

Pablets — Mon, 20 Jul 2026 02:18:41 +0000

From MCP tool design to a durable XState + DBOS guardrail, end-to-end identity with Keycloak, crypto-shredded PII, and an eventing backstop — everything you need to let an AI agent near real money. And it all runs fully offline on your laptop.

AI agent demos are easy. An LLM, a handful of tools, a chat box — twenty minutes and you're taking screenshots.

Then you point the same architecture at a banking workflow, and every shortcut turns into an incident report:

The agent retries a failed call and disburses the loan twice.
The LLM gets creative and calls disburse before the customer accepted the terms.
A network blip mid-conversation and the application state evaporates.
The customer's name and email end up in your logs, your traces, and your vector store — right before they invoke their right to erasure.
Anyone who can reach the API can operate on anyone else's loan, because "the agent sends a customer id" is the entire identity model.

The problem isn't the LLM. The problem is treating a probabilistic text generator as if it were a trustworthy client. It isn't one. It's a brilliant, unreliable intern — and the system around it has to be built the way banks have always built systems around unreliable clients: with hard state machines, idempotency, verified identity, and an audit trail.

That's exactly what this guide builds. Not a theory-heavy overview — a real, running system you can clone, boot with one docker compose up, and break on purpose.

To run the demo and every experiment in this article, check out the GitHub repo for full instructions.

What are we going to build?

By the end of this guide, you'll have a fully functional conversational loan-acquisition system: a customer chats with an agent, picks a loan product, gets a simulation with an interactive amortization chart, accepts the terms, and receives a (demo) disbursement — with every step enforced, persisted, verified, and traceable.

The customer experience looks like an AI chat. Underneath, it's a small bank:

Reading the diagram bottom-up:

Customer Browser → Chainlit UI — a chat UI with a real OIDC login (no demo password gate), persisted threads, and Plotly amortization charts.
MCP Host — the LangGraph agent — a Python agent whose tools are the MCP server's tools, with a deterministic loan flow, layered slot-help, a self-improving RAG, and an AgentCore-compatible /invocations API.
MCP Server Layer — loan-mcp — a FastMCP server exposing the loan journey as 8 tools, verifying the caller's identity token and binding the journey to the caller's own party.
Core Banking Services — four small Node services: an orchestrator (XState + DBOS guardrail), a loans-api (system of record, transactional outbox), a stateless rules-api (lending policy), and a party-api (the only place PII exists, encrypted per-party).
Identity Plane — Keycloak — mints the tokens that every downstream hop independently re-verifies.
Data · Eventing · Key Custody — six isolated per-service PostgreSQL instances (database-per-service), LocalStack SNS/SQS for domain events, and OpenBao Transit for crypto-shredding.

The original article has an interactive, scroll-driven version of this diagram that lights up each layer as you read.

Here's what you'll walk away with:

A pattern for putting a deterministic guardrail between an LLM and anything irreversible
An MCP tool surface designed for agents — docstrings as prompts, errors as data, ordering enforced server-side
Retry + idempotency done properly, including the failure mode most systems ignore
A transactional outbox → SNS → SQS projection backstop, with a DLQ
End-to-end identity: OIDC login, token forwarding, and independent re-verification at three hops
Right-to-erasure by crypto-shredding — PII that can actually be forgotten
A fully offline stack — deterministic fake LLM, LocalStack instead of AWS — that still runs 77 end-to-end scenarios, chaos included

The core principle: the LLM proposes, the machine disposes

Before any code, one design decision drives everything else in this system:

The LLM never decides what is legal. It only proposes. A pure, deterministic state machine — persisted durably — is the single authority on what may happen next.

Every "agent gone rogue" story is a violation of this principle. If your agent can call disburse and the only thing preventing an out-of-order disbursement is the prompt — you don't have a workflow, you have a suggestion.

In this system the loan lifecycle is an explicit XState machine:

export const LoanStateSchema = z.enum([
  'selecting_product',
  'offering_shown',
  'collecting_inputs',
  'simulated',
  'terms_accepted',
  'disbursed',
  'cancelled',
  'failed',
]);

// The loan-acquisition lifecycle as a PURE, deterministic XState machine. It is the
// guardrail: given the current durable state and a candidate event (proposed by the
// non-deterministic LLM, or projected from a loans-api domain event), it answers
// "is this transition legal?" and "what is the next state?". It performs NO side
// effects — everything depends on the machine; the machine depends on nothing.

The machine is pure — no I/O, no context, no side effects. The orchestrator persists the current state durably (DBOS on Postgres), and every command the agent proposes is checked against the machine before anything executes. An out-of-order call — say, disburse_loan while the state is still simulated — doesn't corrupt anything. It comes back as a clean 409 STATE_TRANSITION error that the agent can read, explain, and recover from conversationally.

That one boundary changes the whole risk profile: the LLM can be as creative as it wants, because creativity bounces off the guardrail.

The MCP tool surface: designing tools for an unreliable caller

The agent reaches the banking core exclusively through loan-mcp — a FastMCP server exposing 8 tools that map 1:1 to the orchestrator's REST edge. This layer looks thin, but it's where most of the agent-facing design lives.

mcp = FastMCP("loan-acquisition", host=MCP_HOST, port=MCP_PORT)

@mcp.tool()
def get_loan_products() -> dict[str, Any]:
    """List the loan products the customer can choose from (the catalog).

    Call this FIRST, at the very start of a loan conversation, to present the
    available products (e.g. personal / auto / mortgage) as discrete choices. Each
    product carries the limits and installments the later steps must respect.

    Returns:
        {"products": [ <product>, ... ]} on success, each product shaped like
        {"id", "label", "currency", "minAmountMinor", "maxAmountMinor",
         "allowedInstallments": list[int], "baseRateBps", "disclosure"}
        (amounts in minor units, e.g. cents), or {"error": <RFC 9457 problem>} if
        the rules service is unavailable.
    """
    return _run_rules("get_loan_products",
                      lambda: {"products": _rules_client.get_products()})

Three deliberate choices are packed in here:

Docstrings are prompts. The LLM reads these to decide when and how to call each tool. "Call this FIRST" isn't documentation flavor — it's behavioral steering that costs zero tokens of system prompt. Every one of the 8 tools carries ordering hints, argument semantics, and the exact response shape.

Errors are data, not exceptions. A failed call returns {"error": <RFC 9457 problem>} instead of raising. An exception dies somewhere in the MCP transport; a structured error reaches the model, which can then tell the customer "that amount is above the limit for this product" instead of stalling. The orchestrator's 409 guardrail rejections ride the same channel.

Money is integers. Every amount is minor units (amount_minor, cents) end to end. LLMs are bad at floating point; ledgers are worse at it.

The module docstring states the decomposition contract explicitly — this file knows nothing about HTTP:

"""
Secret this module hides: the MCP tool surface — tool names, docstrings, typed
signatures, and how an OrchestratorError becomes a usable {"error": {...}} the
agent can read. It knows NOTHING about httpx, paths, or the HTTP contract;
those are orchestrator_client.py's and rules_client.py's secrets. The only
reason to change this file is a change to the agent-facing tool shape.
"""

When the tool shape changes, one file changes. When the HTTP contract changes, a different file changes. That's the whole game.

The end-to-end flow

One full journey, numbered:

Login — The customer signs in through Keycloak (a real authorization-code flow — more on this below) and opens the chat.
Chat turn — The UI POSTs to the agent's /invocations endpoint with the customer's access token.
Catalog — The agent calls get_loan_products → get_offering, presenting products and terms as discrete choices.
Slot filling — The LangGraph flow collects amount and installments, pre-validating against the offering's limits so out-of-policy requests get caught conversationally.
Simulation — simulate_loan mints the application (the Control Record) via the orchestrator; the UI renders the payment plan plus an interactive amortization chart.
Guardrailed progress — accept_terms, then disburse_loan — each command checked against the XState machine, executed against loans-api with retry + idempotency, then projected back into durable state.
Events — loans-api's outbox publishes domain events to SNS; the orchestrator's SQS consumer converges the projection even if a direct response was lost.

Note: the LLM may loop between steps 3–6 as the conversation meanders — customers change amounts, ask questions, go back. That's fine because ordering is enforced server-side. The conversation is free; the ledger is not.

Surviving failure: bounded retries on top of idempotency

Here's the failure mode that separates toy agents from production systems: the orchestrator calls loans-api, loans-api commits the write, and the response is lost — connection reset after commit. Did the disbursement happen? The caller cannot know.

If you retry blindly, you double-disburse. If you never retry, every network blip strands a customer. The answer is a pair of mechanisms that only work together:

1. Every logical command carries an idempotency key, and loans-api dedupes on it:

The key is built once per logical command — a retry replays the same key.
loans-api's runIdempotentWrite records the response under that key inside the same transaction as the domain write.
A replayed key returns the recorded response. Never a second write.

2. The orchestrator retries only what's worth retrying, on a bounded schedule:

// Transient = a fault a RETRY could plausibly clear: a 5xx UpstreamError or a bare
// transport failure (connection reset / timeout). Terminal = any 4xx DomainError
// (validation, not-found, illegal transition): retrying only burns patience.
export function isTransient(error: unknown): boolean {
  if (isDomainError(error)) {
    return !(error.status >= 400 && error.status < 500);
  }
  return true;
}

// maxAttempts 5, 1s base, ×2 growth: retries at ~1s, 2s, 4s, 8s → ~15s of backoff.
// Jitter is ADDITIVE (layered on top, never subtracted): it decorrelates concurrent
// retriers but cannot shrink the total below the ~15s floor.
export const DEFAULT_RETRY_SCHEDULE: RetrySchedule = {
  maxAttempts: 5,
  baseDelayMs: 1_000,
  backoffFactor: 2,
  jitterMs: 250,
};

The comment in the real code says it plainly:

// PRECONDITION: `op` MUST be idempotent. A retry may replay a call whose earlier
// attempt already committed downstream but whose response was lost (the ambiguous
// request-landed-response-lost failure). Every loans-api call satisfies this.

Two details worth stealing:

The retry policy is a plain injected decorator (retrying-loans-client wraps the loans client). The command service doesn't know retries exist. Unit tests inject fake timers and fake randomness; production injects nothing and gets the wall clock.
The chaos tests prove the retries ran. The e2e suite uses Toxiproxy to inject a persistent fault and then asserts elapsed > 12s — if the backoff schedule silently stopped running, the test fails. That's also why the jitter is additive rather than "full jitter": full jitter can collapse a delay toward zero and would make that proof flaky.

The eventing backstop: outbox → SNS → SQS

The synchronous path can still lose a response after all retries. For that, the system keeps a second, asynchronous road to consistency:

The sequence has three phases:

Command — loans-api writes the domain change and an outbox row in one transaction. There is no window where the state changed but the event doesn't exist.
Async relay — a DBOS-durable poller publishes unpublished outbox rows to an SNS topic (loan-domain-events) and marks them published. At-least-once, by design.
Projection backstop — an SQS queue subscribed to the topic feeds the orchestrator's consumer, which applies events to its projection idempotently per event id. Three failed receives and the message redrives to a DLQ instead of poisoning the queue.

Backstop, not source of truth. State is durable in loans-api regardless. The event path exists so the orchestrator's view converges even when a direct response was lost — and duplicates are absorbed by idempotency on both sides.

Locally, SNS and SQS are LocalStack — the offline stack exercises the real AWS SDK code paths, queue redrive policy included, without an AWS account.

Identity end to end: the agent is just another untrusted client

Most agent demos handle identity like this: the UI tells the agent who the user is, and everyone downstream believes it. In a loan flow, that's a vulnerability, not a shortcut.

This system treats the access token as the only identity carrier, and re-verifies it independently at every hop that matters:

Phase 1 — Login. The Chainlit UI runs a real authorization-code flow against Keycloak with prompt=login (logout must actually log out — an SSO cookie that silently waves you back in is wrong for a loan flow). One subtlety worth knowing: the browser reaches Keycloak at https://localhost:8543 while backends reach it in-network at https://keycloak:8443 — the classic two-hostname split. The token's issuer is pinned to the frontend URL; the JWKS is fetched in-network. Chainlit's stock provider assumes one base URL for both, so the UI registers a small custom provider that splits them.

Phase 2 — Every chat turn. The UI forwards the access token to the agent; the agent forwards it to loan-mcp; and then the verification cascade begins:

def verify(self, token: str) -> VerifiedPrincipal:
    """Prove the token authentic, then return its principal; raise TokenError otherwise.

    Verifies (in order): a signing key exists in JWKS for the token's kid; RS256
    signature; `exp` (with leeway); `iss` equals the configured issuer. Then enforces
    the app policy: the email must be present AND Keycloak-verified, and — if
    configured — the authorized party (azp) must match.
    """
    signing_key = self._keys.get_signing_key_from_jwt(token)
    claims = jwt.decode(
        token, signing_key.key,
        algorithms=_ALGORITHMS, issuer=self._issuer, leeway=self._leeway,
        options={"verify_aud": False, "require": ["exp", "iss"]},
    )

loan-mcp verifies the token against Keycloak's JWKS and binds the journey to the token's party — the verified email resolves (via blind index) to a party reference in the Party Directory. With LOAN_MCP_AUTH_ENFORCED=true, a simulate_loan without a verifiable token is refused outright. Fail closed.
The orchestrator re-verifies the same token independently and confirms the token-derived party matches the customerRef being operated on. This is the defense-in-depth move: a caller who bypasses loan-mcp entirely and hits the REST edge with a valid token for a different party is rejected here too.
party-api verifies tokens on its one irreversible route — erasure — requiring the identity to be the party itself or a compliance-officer role.

Trade-off callout: verify-when-present vs require. The orchestrator runs ORCH_AUTH_MODE=verify-when-present: a present token is verified and ownership-checked (fail-closed on mismatch), but token-less callers still pass — which keeps the direct-REST e2e suite and internal tools working against the same stack. A deployment without that path should run require. The point is that the mode is an explicit, named decision — not an accident of which middleware happened to be installed.

PII that can actually be forgotten

Loan conversations are PII magnets — and LLM systems multiply the copies: logs, traces, checkpoints, embeddings. This stack's answer is architectural, not procedural:

Only one service ever holds PII. The party-api — modeled on the BIAN Party Reference Data Directory service domain — stores identity data; every other service holds an opaque partyReference and copies nothing. Names never enter the orchestrator's database, the agent's checkpoints, or the loan records.

And that one service can't read its own database. PII is stored as ciphertext under a per-party OpenBao Transit key; lookups run on blind indexes (HMACs), not plaintext. Right-to-erasure is then a two-step:

Destroy the party's Transit key.
Delete the rows.

Step 1 is the interesting one: destroying the key makes every copy of that ciphertext — live rows, WAL, backups — undecryptable at once. That's crypto-shredding: erasure that doesn't require rewriting audit history or hunting down replicas. No other service needs to participate, because no other service ever had the data.

On the agent side, the RAG that learns from customer conversations scrubs PII before anything is embedded or stored — the vector store learns problems and solutions, not people.

When the customer gets stuck: layered help, self-improving RAG

A deterministic slot-filling flow has a classic failure mode: the customer answers "well, it depends…" and the flow re-asks the same question forever. This system layers three escapes:

Deterministic guidance first — per-slot help text, constraints, examples. Zero LLM cost, always available.
A sandboxed LLM helper — a constrained chain that answers the customer's actual question about the slot, without being allowed to drive the flow.
A self-improving RAG — resolved question→solution pairs (PII-scrubbed) are embedded into pgvector; the next stuck customer with a similar question gets the learned answer. A second, curated knowledge_docs collection answers conceptual questions ("how does French amortization work?") from bundled explainer documents.

Everything fails soft: if pgvector is unavailable, guidance degrades to the deterministic layer — a help subsystem outage never takes down the loan flow.

One production bug from this feature is worth retelling, because it's invisible until you stream. The helper chains run inside the flow node, and the node appends the final message itself — so the helper's tokens were streaming to the UI and the final message arrived afterward: every answer appeared twice. The fix is a tagging pattern:

# Tagged as an internal helper: this runs INSIDE loan_flow_node, which appends the
# final message itself, so the streaming layer drops this chain's token stream.
return (prompt | model).with_config(tags=["loan_flow_internal"])

…and the streaming layer filters spans carrying the tag. If you compose LLM calls inside LangGraph nodes and stream with stream_mode="messages", you will hit this.

Running it: fully offline, deterministically

The entire stack boots with one command and zero cloud dependencies:

docker compose -f loan-acquisition-infra/compose/docker-compose.yml up --build
# → https://localhost:8000  (accept the demo CA once)

Concern	Cloud deployment	Offline compose stack
LLM	Amazon Bedrock	`LLM_PROVIDER=fake` — deterministic scripted brain
Embeddings	Provider embeddings	Deterministic hash embeddings (no network)
SNS / SQS	AWS	LocalStack (same SDK code paths)
Identity	Managed IdP	Keycloak container, real OIDC flows
Secrets / keys	Cloud KMS	OpenBao Transit (real server mode, raft)
TLS	ACM certs	Local demo CA (`compose/tls`) on UI, Keycloak, OpenBao, party-api
Postgres	RDS (one per service)	Six isolated Postgres 16 instances — database-per-service (+ pgvector on the agent's, hypopg, pg_stat_statements)

The deterministic fake LLM is the unsung hero: it drives the same create → offer → simulate → disburse journey every time, which is what makes 77 Cucumber end-to-end scenarios — including durability chaos with Toxiproxy latency/faults and WireMock — reliable enough to run on every change. The agent itself is AgentCore-compatible (/invocations contract), so the same code targets AWS when you want the cloud.

For day-two visibility, pgweb gives you a free window into the DBOS workflow tables and the outbox — you can literally watch a retry schedule execute and an outbox row flip to published.

Conclusion

We went from "an LLM with tools" to a system where the agent is treated as what it really is: a persuasive but untrusted client, wrapped in the same discipline banks have always applied — a deterministic state machine as the single authority, idempotency underneath every retry, identity verified at every hop instead of asserted once, PII isolated and crypto-shreddable, and an event backstop for the failures the happy path can't see.

The core takeaway is simple: you don't make an agent safe by prompting it better — you make it safe by building a system that stays correct when the agent is wrong. The LLM proposes; the machine disposes. Everything else in this architecture is that one sentence, applied layer by layer.