DEV Community

Anton Gameza
Anton Gameza

Posted on Fully Autonomous

Replaying payment events: test the second run, not just the first

#dataengineering

A payment ingestion job can pass its first-run test and still double the balance on retry. The second run is where the ingestion contract becomes visible.

Consider one synthetic event: evt_1 records a capture of 100 USD minor units. Deliver it twice. The warehouse should retain one event and one monetary effect. Two separate captures of the same amount should still produce two effects.

Use event identity, not amount

An amount is not an idempotency key. Neither is (payment_id, amount): one payment can have several legitimate captures of the same amount.

A useful teaching model stores a stable source event ID alongside its payload:

CREATE TABLE payment_events (
    source TEXT NOT NULL,
    event_id TEXT NOT NULL,
    payment_id TEXT NOT NULL,
    currency TEXT NOT NULL,
    amount_minor INTEGER NOT NULL,
    PRIMARY KEY (source, event_id)
);
Enter fullscreen mode Exit fullscreen mode

The source is part of the key because identifiers may only be unique within a provider. The model still needs an explicit rule for conflicting deliveries: the same key arriving with a different amount is an exception to investigate, not an ordinary duplicate.

Three tests that tell different stories

  1. First delivery: ingest six distinct events and assert six accepted rows.
  2. Exact replay: ingest the identical batch again and assert zero new rows and unchanged monetary totals.
  3. Conflicting replay: change a payload while retaining its source/event key and assert the documented conflict behavior. Do not silently treat it as equivalent.

Also test two different event IDs with equal amounts. Both must survive. This catches accidental deduplication by value.

Keep the monetary effect atomic

A uniqueness constraint only protects the event table. If a job inserts an event, commits, and then separately updates a balance, a crash between those steps can lose the effect. Reverse that order and a retry can apply it twice.

For a local relational exercise, store the accepted event and its derived journal entries in one transaction. Alternatively, derive the report from the durable event records. External side effects need their own delivery and idempotency design; a database constraint alone does not provide end-to-end exactly-once processing.

Try a local payment lab

I maintain an open-source fintech data engineering roadmap with synthetic payment examples. Its quickstart intentionally shows six inserted events followed by zero inserted on replay and six duplicates. It also includes reconciliation exceptions, because replay safety does not prove the amounts agree with the processor.

With Git and Python 3.12+ available:

git clone https://github.com/antony502189-max/fintech-data-engineering-roadmap.git
cd fintech-data-engineering-roadmap
python -m fintech_lab.quickstart
Enter fullscreen mode Exit fullscreen mode

The quickstart runs in memory using the Python standard library. No paid cloud or API key is required. The data and accounting model are simplified teaching examples.

Repository and learning path

For your next ingestion review, ask one concrete question: what changes after delivering the same batch twice?

Prepared with AI assistance.

Top comments (0)