DEV Community

Cover image for AI Agents Are Making an Old Distributed Systems Bug More Expensive
AdmilsonCossa
AdmilsonCossa

Posted on

AI Agents Are Making an Old Distributed Systems Bug More Expensive

#ai #distributedsystems #systemdesign #fintech

The dual-write problem existed long before LLMs. Now a duplicate event can trigger fraud investigations, payment holds, refunds, or financial execution.

At 11:57 PM on Black Friday, a global commerce platform approves a €2.4 million payout to a merchant.

The ledger commits the transaction. The application publishes PayoutApproved. Then the request times out.

Nothing is necessarily down. The database is healthy. The broker may also be healthy. The network recovered milliseconds later. Every dashboard is still green.

Yet the system has lost something more dangerous than availability.

It has lost certainty.

Sequence diagram showing a payment service committing a payout to the ledger database successfully, then publishing a PayoutApproved event to the broker. The acknowledgement times out, leaving the payment service unable to determine whether the event was received — creating a risk of either a missed risk review (if not retried) or a duplicate delivery (if retried).

Did the broker receive the event?

If the producer does nothing, the risk system may never analyze the payout. If the producer retries — and the first publication actually succeeded — the event may be delivered twice. If downstream execution is not idempotent, the same business operation may be applied twice.

A few milliseconds of uncertainty can become a multimillion-euro question:

Did the operation fail, or did we merely fail to observe that it succeeded?

This is not fundamentally a Kafka problem, a NATS problem, or an AI problem. It is a distributed systems problem.

Modern AI agents make it more expensive because event consumers are becoming more capable. A duplicate event no longer has to mean a duplicated projection or a repeated log line. It may wake an agent that assesses fraud, recommends a payment hold, launches an investigation, requests a refund, or calls a tool connected to an external financial system.

The delivery semantics did not get worse. What sits behind the event became more powerful.

The architecture needs answers to five questions:

  • Where does the event become authoritative?
  • How is publication recovered after failure?
  • How do consumers tolerate redelivery?
  • How is ordering preserved where the business requires it?
  • How do we prove that an irreversible external effect happened once?

1. Your Broker Does Not Solve the Dual-Write Problem

A service that approves a payout typically performs two operations that conceptually belong together but live in two separate transactional systems: the database and the broker.

Flowchart showing 'Approve Payout' branching into two separate operations: 'Update Database' and 'Publish PayoutApproved', connected by a dashed line labeled 'different transaction boundary,' illustrating that the two writes are not atomic.

There is no automatic atomic transaction spanning both systems, which creates three distinct failure states:

Failure Consequence
Database commits, event publish fails Business state exists, downstream systems never learn about it
Event published, database transaction rolls back Consumers observe something that never became true
Broker accepts event, ACK is lost, producer retries Possible duplicate delivery

This is the dual-write problem. A broker gives you decoupling, buffering, durability, fan-out, replay, and failure isolation. What it does not give you is atomicity between an event and the database state that caused that event to exist. That distinction is the foundation for everything that follows.

2. Kafka, Redpanda, RabbitMQ, and NATS Solve Different Problems

The right question isn't "which broker is best?" — it's "which guarantees does this workload actually require?"

  • Apache Kafka — strong for long retention, partition-based ordering, high throughput, replay, stream processing, and a mature connector ecosystem. Since Kafka 4.0, ZooKeeper mode is gone and KRaft is mandatory — one less operational dependency, but partitioning, retention, rebalancing, and DR still require deliberate engineering.
  • Redpanda — broad Kafka protocol compatibility, including idempotent producers and transactions, with a different operational profile. The architectural boundary is unchanged: a Redpanda transaction cannot make a write to an external Postgres database — or a bank transfer — atomic.
  • RabbitMQ — quorum queues, publisher confirms, persistent messages, Single Active Consumer, streams, and routing exchanges make it a strong fit for commands, task distribution, and workflow orchestration.
  • NATS JetStream — combines low-latency NATS messaging with persistence, durable consumers, retention, and acknowledgement-based delivery. Compelling for distributed topologies, request-reply workloads, and edge environments.
Dominant requirement Commonly suitable technology
Long retention, replay, stream processing Kafka or Redpanda
Low-latency messaging, request-reply, distributed topologies NATS JetStream
Work queues, commands, complex routing RabbitMQ
Large Kafka ecosystem and connector estate Kafka or Redpanda
Legacy database change capture Debezium + a suitable broker

None of them eliminates the dual-write problem by itself.

3. AI Agents Don't Create the Problem — They Increase Its Blast Radius

The dual-write problem predates LLMs by decades. Retries, duplicate messages, lost acknowledgements, and reordered events are established distributed systems failure modes. What's changed is what a consumer can now do with a duplicate.

Flowchart showing a linear, low-risk path: PayoutApproved event flows to a Projection Consumer, which updates a dashboard.

Flowchart showing a higher-risk path: PayoutApproved event flows to an AI Risk Agent, which produces a Fraud Recommendation, leading to a Policy Decision and then a Payment Action, both the AI Risk Agent and Payment Action nodes are highlighted in red to indicate increased blast radius.

If the old projection consumer processes a duplicate twice, and the projection is idempotent, the impact may be negligible. If the same duplicate now traverses an agentic path, it can trigger a second fraud investigation, a second account block, or a second refund instruction.

That doesn't make AI agents inherently unsafe. It means the architecture around them must assume messages arrive more than once, agents can be wrong, tools can time out, models can change, and external systems can return ambiguous outcomes.

An AI agent should never move money simply because it generated a plausible answer. The safer boundary separates reasoning from execution:

Flowchart showing PayoutApproved triggering an AI Risk Agent, which outputs PayoutRiskAssessed to a Policy and Approval Service. The policy service branches to either ReleasePayout (within limits) or ManualReviewRequired (needs review), both feeding into an Idempotent Payment Executor that writes to the Ledger.

AI recommends. Policies decide. Systems execute.

The agent analyzes, classifies, explains, and recommends. A deterministic service evaluates limits, permissions, segregation of duties, and compliance. Only then does an idempotent executor perform the external effect.

4. CDC Is More Than a Legacy Escape Hatch

Enterprise systems rarely start clean. Banks, insurers, and telecoms often depend on ERPs and settlement engines built years before event-driven architecture became mainstream. Rewriting them to emit domain events may be too risky.

Change Data Capture (commonly Debezium) observes transactional database logs and captures committed changes without touching the legacy application.

Flowchart showing a Legacy ERP database with a PAYMENT_BATCH table feeding into Debezium via log-based CDC, then into a Translation / Anti-Corruption Layer, which outputs three stable domain events: PayoutApproved, BeneficiaryChanged, and SettlementCompleted.

The trap: a row mutation is not automatically a domain event. STATUS = 'A' might mean approved, awaiting review, or an internal transition with no meaning outside the legacy system. If downstream services must understand physical tables and cryptic status codes, the database schema has silently become a public contract — and a fragile one.

A translation layer protects the rest of the organization from the legacy system's physical representation. CDC is also strategic beyond migration: it can feed search indexes, warehouses, read models — and most importantly, it can capture an Outbox table, transporting events the application deliberately designed rather than technical noise.

5. Transactional Outbox Fixes the Birth of the Event

A new payment service knows two facts at the same instant: the payout changed state, and an event describing that change must eventually be published. Instead of two independent writes, it records both facts in one local transaction.

BEGIN;

UPDATE payouts
SET status = 'APPROVED', version = 17
WHERE id = 'pay_7f9q';

INSERT INTO outbox (
    event_id, aggregate_id, aggregate_version,
    event_type, payload, occurred_at
) VALUES (
    'evt_01K...', 'pay_7f9q', 17,
    'PayoutApproved', '{...}', CURRENT_TIMESTAMP
);

COMMIT;
Enter fullscreen mode Exit fullscreen mode

Flowchart showing a Business Transaction entering a single database transaction boundary containing 'Update payout row' and 'Insert Outbox event,' both feeding into a COMMIT step, followed by a Relay or CDC process reading the Outbox and publishing to the Broker.

If the transaction fails, neither the new state nor the event exists. If it commits, both exist. Publication becomes a recoverable asynchronous process rather than a second point of failure. This solves the origin problem — the event is born atomically with the state that justifies it. It does not solve every delivery problem.

6. Outbox Does Not Mean Exactly-Once

Sequence diagram showing a Relay reading a pending event from the Outbox table, publishing it to the Broker, and receiving an acknowledgement. The relay then crashes before marking the event as published. After restarting, the relay reads the same pending event again and publishes it a second time, illustrating an at-least-once duplicate that results from safe crash recovery rather than a broker fault.

The duplicate here isn't evidence of a broken broker — it's the correct consequence of recovering safely from uncertainty.

Do not build correctness on the assumption that delivery happens exactly once. Build consumers so repeated delivery does not repeat the business effect.

The working invariant:

Record atomically once. Deliver at least once. Apply the effect once.

Kafka and Redpanda offer exactly-once semantics inside defined transactional boundaries. NATS offers publication deduplication within a configured window. None of that automatically makes POST https://external-bank/pay exactly-once — the broker doesn't control the bank.

7. Idempotency Starts Before the Broker

Outbox solves one form of duplication, not all of them. If a client retries POST /payouts after a lost response, the server may create two payouts before Outbox ever becomes relevant.

Flowchart showing a client POST request with an Idempotency-Key header flowing through Ingress dedup, then Business Transaction plus Outbox, then the Broker, then Inbox dedup using an ON CONFLICT DO NOTHING pattern, and finally an Idempotent Consumer.

Every event carries an immutable eventId. A consumer records that ID before applying its effect:

BEGIN;

INSERT INTO inbox(event_id, received_at)
VALUES ('evt_01K...', CURRENT_TIMESTAMP)
ON CONFLICT DO NOTHING;

IF event_was_inserted THEN
    UPDATE payout_projection ...;
END IF;

COMMIT;
Enter fullscreen mode Exit fullscreen mode

If evt_01K... arrives again, the Inbox constraint blocks the repeated local effect. One boundary still remains: the external world.

8. Idempotency Must Reach the Final Side Effect

Internal deduplication means nothing if the last hop — the actual bank transfer — lacks a uniqueness guarantee.

Flowchart showing an event ID flowing into a Payment Executor, which calls an External Provider using the event ID as an idempotency key. The response branches into Success, updating the ledger to PAID, or Timeout/Ambiguous, setting status to UNKNOWN and triggering a Reconciliation Job that loops back to check the response again.

A timeout means UNKNOWN, not FAILED. Without uniqueness at the destination, an idempotency contract with the provider, or a reconciliation loop, "exactly-once" is not a guarantee — it's optimism.

9. Ordering Is a Business Property, Not a Broker Marketing Feature

"Messages are ordered" is the wrong claim. The real question: ordered with respect to what?

Diagram contrasting the intended event order (v17 PayoutApproved, v18 PayoutHeld, v19 PayoutReleased) with an actual delivered order where v19 PayoutReleased arrives before v18 PayoutHeld, both flagged with warning symbols to show an invalid resulting state.

Processing v17 → v19 → v18 can produce an invalid state even though every message eventually arrives. Event envelopes should carry an aggregateVersion so consumers can detect stale events, duplicates, and sequence gaps.

A NATS subject is a routing address; a Kafka partition is an explicit storage and ordering boundary — they are not equivalent. Deterministic routing in NATS needs an opaque entity key mapped consistently to a shard:

finance.payouts.eu.s042.pk_7f9q.approved
Enter fullscreen mode Exit fullscreen mode

Flowchart showing an opaque payout key hashing to Shard s042, routed to a Serial Worker or Consumer, which checks the aggregateVersion. If events are in order, the effect is applied; if a gap is detected, the event is held in a reorder buffer.

Parallel workers preserve arrival order, not completion order — if external side effects depend on order, storage ordering alone cannot protect the invariant.

10. Edge and Multiregion Systems Make Authority Explicit

A branch, regional gateway, or edge deployment that must survive a disconnected network needs a local database and local Outbox that synchronizes once connectivity returns.

Flowchart showing an Edge domain with a local database feeding a local Outbox, separated from a Central global event platform by a red 'network down, sync blocked' node, which resolves once connectivity is restored and synchronizes to the central platform.

A NATS leaf node does not automatically turn a local database into a conflict-free offline store. The architecture must still define: which system is authoritative per entity, how conflicts resolve, how gaps are detected, what happens when local disk fills, and how reconciliation completes.

Local availability with eventual global convergence — not global consistency independent of the network.

11. The Reference Architecture

Large flowchart showing two parallel tracks. The Legacy Systems track: ERP or Core Ledger to CDC to Translation Layer to Stable Domain Events. The New Services track: Payment Service to a single local transaction containing Business Tables and Outbox, feeding a Relay or CDC Router. Both tracks converge on a central Broker (Kafka, Redpanda, NATS JetStream, or RabbitMQ), which fans out to Ledger Projection, AI Risk Agent, Audit Service, and Notification Service. The AI Risk Agent produces PayoutRiskAssessed, feeding a Policy and Approval Service, then an Idempotent Payment Executor, then the Ledger, then an External Provider, then Reconciliation.

The essential building blocks: an authoritative transactional source, an atomic Outbox, reliable publication, durable transport, immutable event identity, idempotent consumers, ordering where the business requires it, deterministic policy boundaries, idempotent external execution, and reconciliation. The broker is one component inside it — not the architecture itself.

12. Event Envelopes Are Part of the Contract 

{
  "eventId": "evt_01K...",
  "eventType": "PayoutApproved",
  "aggregateId": "pay_7f9q",
  "aggregateVersion": 17,
  "schemaVersion": 3,
  "occurredAt": "2026-08-07T10:57:01Z",
  "correlationId": "corr_8x7b",
  "causationId": "cmd_9c3a"
}
Enter fullscreen mode Exit fullscreen mode

eventId establishes immutable identity. aggregateVersion enforces sequence correctness. correlationId and causationId connect a business flow end-to-end. This metadata becomes essential once AI agents participate.

13. AI Decisions Need Evidence, Not Just Logs

"Payout blocked" is not an adequate audit trail. A serious system must be able to reconstruct the full decision path:

Flowchart showing a linear evidence chain: Event to Risk Agent to Evidence Gathered to Model and Version to Risk Assessment to Policy Version to Approval State to Final Deterministic Decision to Executed Action, with Model/Version and Final Decision nodes highlighted to emphasize the audit trail.

The AI output is evidence in the decision process — it should not automatically become the source of financial truth. The ledger, the policy service, and the permissions model remain authoritative. Governance defines minimization, hashing, retention, and access control — the goal is reproducibility and accountability, not surveillance.

14. Observability Should Measure Correctness, Not Just Availability

A dashboard showing green CPU and green uptime can completely miss the failure from the opening scenario — nothing was unavailable, the system was uncertain.

Track: CDC lag, publication lag, consumer lag, redelivery rates, duplicate detection counts, Inbox conflicts, sequence gaps, dead-letter growth, Outbox backlog, reconciliation differences, agent failures, policy rejections, and external timeout rates.

The most dangerous state isn't service = DOWN. It's:

payment = UNKNOWN
Enter fullscreen mode Exit fullscreen mode

That deserves first-class visibility as a reconcilable business object — not a line inside a generic error counter.

15. Security Must Survive the Event Pipeline

Event systems copy information widely, which makes careless identifiers expensive. Don't embed account numbers, card details, or national IDs in topic/subject names — they leak into logs, metrics, traces, and dashboards.

❌  finance.account.447819002343.payment
✅  finance.payouts.eu.s042.pk_7f9q.approved
Enter fullscreen mode Exit fullscreen mode

Encryption in transit and at rest, fine-grained access control, schema governance, controlled replay, dead-letter handling, and data residency controls are non-negotiable. Durability without governance only makes mistakes survive longer.

16. Don't Turn Broker Choice Into Religion

Scenario Recommended approach
Legacy system that cannot be modified CDC, translation layer, stable domain contracts
New transactional service Transactional Outbox + relay or CDC-over-Outbox
Commands and external financial effects Idempotency at ingress, Inbox on consumption, reconciliation
Stream processing, long retention, massive replay Kafka or Redpanda
Low latency, request-reply, edge topologies NATS JetStream
Work queues and enterprise routing RabbitMQ
Offline / multiregion operations Local DB + Outbox + explicit synchronization
AI agents in critical processes Versioned events, policy service, audit evidence

The architecture should follow the invariant. The invariant should not follow the product logo.

17. The Real Meaning of Exactly-Once

"Does this broker support exactly-once?" is rarely the useful question. The useful question is: exactly-once where?

Inside one Kafka transaction? Inside one database? Across your broker and Stripe? Across your application and a bank? Across two regions during a partition? The wider the boundary, the harder the phrase is to defend.

Record the business intent once.
Deliver notifications at least once.
Detect repeated delivery.
Apply each business effect once.
Reconcile anything whose outcome remains uncertain.
Enter fullscreen mode Exit fullscreen mode

Exactly-once is not merely a transport property. Business correctness is an end-to-end property.

18. What AI Changes

AI does not repeal distributed systems. Agents still run on networks. Tools still time out. Databases still commit independently. Messages can still be duplicated.

Two side-by-side flowcharts. 'Before': Event to Consumer to Database Projection. 'Now': Event to Agent to Reasoning to Tool Selection to Policy to External Action, illustrating the increased distance between information and action in agentic systems.

What AI changes is the distance between information and action. As agentic systems gain operational authority, distributed systems correctness becomes more important, not less. Agent reliability is not primarily a prompt engineering problem — the guarantees that matter most still live in atomicity, idempotency, ordering, authorization, isolation, auditability, and reconciliation. The model is only one component.

Conclusion

Transactional Outbox solves one of the hardest boundaries in asynchronous systems by recording business state and publication intent inside the same local transaction. CDC or a relay transports that intent toward the broker. The broker provides durable delivery. The Inbox pattern and immutable event identifiers let consumers tolerate redelivery. Aggregate versions preserve business ordering. Idempotency keys protect commands and external operations. Policy services keep probabilistic reasoning separated from deterministic authority. Reconciliation resolves the cases where the network leaves outcomes uncertain.

AI agents do not create these distributed systems problems. They make ignoring them more expensive.

A duplicate event can now travel farther. A retry can activate more capable software. An uncertain result can trigger decisions that affect customers, accounts, infrastructure, or money.

That is why modern agentic architecture should not begin by asking which model is smartest or which broker is fastest. It should begin with the business invariant:

Every financial movement must have an immutable identity, originate from an authorised transactional state, survive publication failures, tolerate redelivery, preserve the required business order, pass deterministic policy controls, execute idempotently, and remain reconcilable and auditable.

Or, simply:

Record once. Deliver at least once. Apply the effect once.

Then ask the question that matters when everything appears healthy and the acknowledgement never arrives:

Can your system prove that the same payment will not leave twice?

Not whether Kafka stayed online. Not whether the database stayed online. Not whether your AI agent produced the right explanation.

Whether the architecture preserved reality when the network stopped being able to tell you what happened.

Technical References

Top comments (0)