DEV Community: oludeleoluwapelumi

I built a free, offline tool to check your transaction logs for one specific silent failure

oludeleoluwapelumi — Wed, 29 Jul 2026 08:48:51 +0000

I built a free, offline tool to check your transaction logs for one specific silent failure

CIF (Chronological Input Failure) has been through a simulation, a mapping against real Apache Fineract defects, and a reconciliation check against real PaySim data. All of that is public. What was missing was a way for someone else to check their own data without having to hand it to me first.

So I built a small command-line tool that does one thing: it checks whether your transaction log's recorded balances actually reconcile with the transactions that supposedly caused them, and flags the ones that don't, unless something in your data already caught it.

It runs entirely on your machine. Nothing gets uploaded, logged, or sent anywhere. You point it at a CSV, it reads it, it prints a report, that's the whole interaction.

What it actually checks
For each row: does old_balance plus or minus the transaction amount equal new_balance? If not, and nothing flags that row as an error already, it gets reported as a candidate anomaly.
That's a narrower test than CIF's original mechanism, which is about validation completing before commit. Most transaction logs don't record validation and commit as separate timestamped events, so this checks something related but smaller: whether the recorded end state is internally consistent with what supposedly produced it.

Why I'm being upfront about its limits
The first time I ran an early version of this exact check against real PaySim data, it flagged 75.92% of transactions. That wasn't a real finding. It took investigating two dataset-specific quirks, untracked merchant balances and balances clipped to zero instead of going negative, plus fixing a direction bug in my own formula, before the number came down to something real: 0.07%, 26 rows, in about 35,000 checkable transactions.
I'm telling you that because if you run this on your own data and get a high number back, that is probably not a discovery. It's much more likely your data has its own quirks the tool doesn't know about yet. Read the flagged rows. Ask why. The full writeup of that process, mistakes included, is linked in the repo.

Try it safely first
There's a small, obviously fake sample file included in the repo specifically so you can see what the tool does before pointing it at anything real.

What I'm actually hoping for
If you run this against your own staging or test data and it flags something real, or if it behaves strangely on your data's specific shape, I want to hear about it either way.

Genuinely, if you try it, even just to say it ran fine and found nothing, tell me. That's useful too, and right now, hearing from real people who actually looked at this matters more to me than the numbers. A false positive you can explain is genuinely as useful to me as a real finding, it's exactly the kind of thing that made the PaySim writeup honest instead of just impressive-looking.

The tool, the code, and everything else CIF has produced so far is here: https://github.com/oludeleoluwapelumi/cif-simulation/tree/main/scanner

Testing CIF's Pattern Against Real Payment Data, and Getting It Wrong Twice Before Getting It Right

oludeleoluwapelumi — Mon, 27 Jul 2026 10:24:20 +0000

CIF (Chronological Input Failure) has so far rested on two things: a simulation, and a retrospective mapping against public Apache Fineract defects. Both are useful, but neither involved running a fresh check on real transaction data myself. This post is that attempt, and it includes the mistakes I made along the way, because those mistakes matter as much as the result.

The dataset
PaySim is a mobile money transaction dataset, built to model the transaction patterns of a real mobile money service. It is not raw production data, and I want to be upfront about that limitation from the start. It is a simulation of a real service, not a live extract from one. I worked with roughly 77,000 real, unmodified rows sampled from the public dataset.

The test
Since PaySim does not contain separate validation and commit events, I could not test the original CIF mechanism directly. Instead I defined a narrower, honest question the data could actually answer: does the recorded balance change for each transaction reconcile with the transaction that supposedly caused it, and if it does not, is anything flagging that.

Mistake one
My first pass found that 75.92 percent of transactions showed a balance mismatch. That number should have been a red flag immediately, and it was. A finding that large, on a first pass, usually means the check itself is wrong, not that a massive failure has been discovered.

Investigating it turned up two real explanations. First, PaySim does not track merchant account balances at all. Every transaction with a merchant destination shows a balance of zero before and after, by design, not because anything failed. Every single one of those rows was an automatic, guaranteed mismatch under my original check, for no meaningful reason. Second, when a withdrawal amount exceeds the available balance, PaySim clips the resulting balance to zero rather than allowing a negative number. My check treated that clipping as a mismatch too, which was also wrong.

Mistake two
After excluding both of those, the mismatch rate was still high, 36.8 percent, which was still too high to be a real finding. The remaining problem was mine. My check assumed money always leaves the origin account, which is true for payments, transfers, and withdrawals, but not for cash-in transactions, where money enters the origin account. I had the direction of the arithmetic backwards for an entire transaction type.

The corrected result
After fixing both dataset artifacts and my own formula error, the genuine mismatch rate across roughly 35,000 cleanly checkable transactions was 0.07 percent, 26 transactions.
Every one of those 26 shares the same specific shape: a transaction is recorded, with an amount, a type, and a timestamp, but the account's balance shows no change at all, as if the transaction never touched it. 23 of the 26 carry no fraud flag or error indicator of any kind. The remaining 3 are flagged as fraudulent, and involve unusually large transfers.

What this does and does not establish

This is a real, specific, reproducible pattern found in real recorded data, and it matches the shape CIF describes: a system recording that something happened while the underlying state silently does not reflect it, mostly without any alert.

It does not establish that this reflects a genuine class of bug in real production payment systems. PaySim is a simulation. I cannot rule out that these 26 rows are an artifact of how PaySim itself was generated, rather than evidence of anything that happens in a live system. That distinction matters, and I am not going to blur it to make the finding sound stronger than it is.

What is next
This still is not a live case study. It is a real check on real, if simulated, data, with an honest and stated limitation. The next real milestone remains the same one from every previous post: a live system, a real team, a genuine finding that was not already known.

Reproducibility
All analysis code, sampling procedures, and filtering logic used in this check are available in the repository. The reported mismatch counts can be reproduced from the same PaySim sample by applying the documented reconciliation rules.

The code and full results are available at https://github.com/oludeleoluwapelumi/cif-simulation/blob/main/validation/PAYSIM_RECONCILIATION_CHECK.md

Mapping CIF's Severity Taxonomy Against Real Apache Fineract Defects

oludeleoluwapelumi — Wed, 15 Jul 2026 08:53:09 +0000

Chronological Input Failure (CIF) started as a simulation. The baseline_simulation.py model in the cif-simulation repo showed that under realistic async delay conditions, commit events can process before their corresponding validation completes, producing a measurable causality violation rate.

A simulation proves a mechanism can occur. It does not prove the mechanism occurs in real systems. This post is an attempt to close part of that gap, not by running CIF against a live system yet, but by checking whether the failure patterns CIF describes already show up in the public issue history of a real, independently built financial platform.

What this is, and what it is not

This is a retrospective taxonomy mapping. I searched Apache Fineract's public JIRA issue tracker, an open source core banking platform used by financial institutions in dozens of countries, for tickets describing duplicate transaction processing, retry driven double execution, and chronological or ordering problems. I read each ticket directly, not a summary, and checked whether it matched CIF's categories.

This is not a case study. Nobody at Fineract used CIF. I did not diagnose these issues. Fineract's own engineers found, discussed, and resolved these tickets, in most cases years before I looked at them. What this shows is that CIF's categories are not arbitrary. They describe failure patterns that an unrelated, production financial system independently experienced and had to solve.

A real case study, where CIF is applied to a live system and produces new findings, is still the next milestone. This is groundwork toward that, not a substitute for it.

Method

I read four Fineract tickets directly on the Apache JIRA tracker, verified their status and description text myself, and checked each one against CIF's severity levels. One ticket I initially flagged as relevant, FINERACT-1876, is discussed separately below, because on close reading it does not actually fit CIF and I am not going to force it to.

The Mapping

FINERACT-2304, Batch retry issue, duplicate records
Link:https://issues.apache.org/jira/browse/FINERACT-2304
Status: Resolved, Fixed

A batch API call with an open transaction could trigger retry logic within that same open transaction, producing duplicate entries. Separately, if a transaction had already finished but an exception still occurred inside the retry boundary, a retry could fire on an already completed operation.

CIF classification: Level 2, Event Replay. This is a direct match, a duplicate write caused by retry logic firing after a completed operation.

FINERACT-1744, System idempotency
Link:https://issues.apache.org/jira/browse/FINERACT-1744
Status: Resolved, Implemented

This is not a bug report. It is a proactive systemic fix. Fineract's team identified that in high availability deployments, retried API operations could execute twice when a prior execution had already completed, and built dedicated infrastructure against it: an Idempotency-Key API header, idempotency handling for external events, and new command statuses (UNDER_PROCESSING, ERROR) to track execution state.

CIF classification: this does not map to a single severity level, and that is the point. It is stronger evidence than any individual bug ticket, because it shows the failure pattern was significant enough for the Fineract team to build permanent architecture against it, rather than patch isolated incidents after the fact.

FINERACT-2188, duplicate repayments reported via mobile app
Link:https://issues.apache.org/jira/browse/FINERACT-2188

Status: Closed, Resolution: Abandoned

A user reported repayments duplicating within milliseconds through a mobile app. This is a reported issue describing symptoms consistent with CIF, not a confirmed root caused bug. Fineract closed it as abandoned, meaning the root cause was never formally established. It could reflect a core platform issue, a client side integration problem, or a duplicated third party callback.

CIF classification: Level 2, Event Replay, symptom consistent, root cause unconfirmed. I am including it because the symptom matches, not because it was proven.

FINERACT-2457, share transaction chronological validation

Link:https://issues.apache.org/jira/browse/FINERACT-2457

Status: Confirmed via linked pull request

Fineract's share transaction validation logic blocked new transactions dated earlier than any existing transaction on the account, including transactions that had already been rejected or reversed. In effect, the system's record of transaction history included events that were no longer financially real, and it enforced ordering rules against that stale history, incorrectly blocking legitimate transactions.

CIF classification: this does not cleanly fit the existing four levels, and I would rather say that plainly than stretch it. It is a genuine chronological integrity problem, the system's understanding of event order did not reflect financial truth, but the mechanism is different from the retry and duplication pattern the four levels were built around. I am noting it as a related but distinct pattern worth folding into a future revision of the taxonomy, not forcing it into a box it does not belong in.

A pattern I checked and ruled out
FINERACT-1876 (https://issues.apache.org/jira/browse/FINERACT-1876) describes a Postgres serialization error, could not serialize access due to concurrent update, occurring intermittently during scheduled job execution. I initially thought this might be a CIF match. On reading the actual ticket, it is not. CIF describes silent divergence, the system reports success while the underlying state has quietly gone wrong, with no alert. FINERACT-1876 is the opposite of that. The database's own isolation guarantees caught the conflict and threw a hard, visible error, the job failed loudly, the test suite failed with it. That is a safeguard working as intended, not a CIF event. I am including this here because being willing to rule something out is part of taking the taxonomy seriously.

What this does and does not establish

It establishes that duplicate execution, stale chronological state, and ordering problems are real, recurring concerns in at least one production grade, independently built financial platform, serious enough in one case to justify dedicated architectural work.

It does not establish that CIF as a diagnostic process finds anything a competent engineering team would not already find on its own. Fineract found and fixed every issue above without CIF. The next real test for CIF is whether applying it to a live system surfaces something that was not already known.

What is next

I am looking for a small fintech engineering team willing to let me run a CIF style pass over a sample of their transaction or event logs, at no cost, with full confidentiality. If that is you, or you know someone it might be, I would like to hear from you.

Full simulation code is in the cif-simulation repository. The complete taxonomy mapping, including links to every source ticket, is at https://github.com/oludeleoluwapelumi/cif-simulation/blob/main/validation/FINERACT_TAXONOMY_MAPPING.md

The CIF Framework: Measuring Ledger Integrity in High-Volume Payment Systems

oludeleoluwapelumi — Fri, 03 Jul 2026 07:06:12 +0000

Modern payment systems report success. But success and truth are not the same thing.

This paper introduces the CIF Framework a measurement and classification system for a specific class of distributed systems failure called Chronological Input Failure. CIF occurs when the chronological integrity of a financial event pipeline breaks down under concurrency and retry pressure, causing ledger state to diverge from financial truth without triggering standard observability alerts.

We introduce three measurable metrics the Causality Violation Rate, the Reconciliation Debt Index, and the Ledger Integrity Score and present simulation results showing a baseline CVR of 8.3% under realistic async payment pipeline conditions.

This is not a theoretical paper. Everything here is reproducible. The simulation is open source. The metrics are computable. The cost model is conservative.

Introduction — The Invisible Failure Mode

There is a class of failure in distributed financial systems that standard monitoring cannot detect.

It does not cause downtime. It does not trigger alerts. It does not show up in your uptime dashboard or your latency graphs.
It shows up three weeks later when your reconciliation team cannot explain why 847 transactions have mismatched ledger states. Or when your fraud model starts flagging legitimate transactions because its training data has been quietly corrupted. Or when your CBN auditor asks you to prove the causal sequence of your transaction history and you cannot.

This failure mode has a name now.

Chronological Input Failure.

The Problem Space — When System Success Does Not Equal Financial Truth
Consider a standard payment transaction lifecycle:

User initiates transfer
Validation hook fires — fraud check, KYC verification, balance check
Validation completes
Commit writes to ledger
User sees success

In a deterministic system this sequence is guaranteed. Step 3 always precedes step 4.

In an async distributed system under network pressure, this guarantee does not hold.

When the validation webhook experiences a 12ms network delay due to partition rebalancing or upstream service latency, the commit event can arrive at the ledger processor before the validation has completed. The ledger writes a result that was never properly validated.

The system reports success. The financial truth is gone.

This is the core of CIF. Not a crash. Not an error. A silent divergence between what the system believes happened and what actually happened.

The CIF Pipeline Diagram

The following diagram shows exactly where CIF emerges in a distributed payment architecture and why it cannot be detected by standard observability tools.

flowchart TD

classDef default fill:#F8FAFC,stroke:#475569,stroke-width:1px,color:#0F172A;
classDef condition fill:#FEF2F2,stroke:#EF4444,stroke-width:1.5px,stroke-dasharray: 4 4,color:#991B1B;

subgraph INTENT["1. INTENT LAYER: Logically Required Sequence"]
A["Original Transaction Sequence: t1 → t2 → t3"]
end

subgraph IDEAL["2. DETERMINISTIC MODEL: Assumed Behavior"]
B1["Event Processing and Commit Layer\nPreserves Timeline: t1 → t2 → t3"]
C1["Consistent Ledger State\nSYSTEM SUCCESS == FINANCIAL TRUTH"]
B1 --> C1
end

subgraph REAL["3. DISTRIBUTED EXECUTION: Observed Behavior"]
B2["Distributed Processing Layer\nNon-deterministic order: t2 → t1 → t3"]
R1{"Network Delay / Retry Loop"}
C2["Diverged Ledger State\nSYSTEM SUCCESS != FINANCIAL TRUTH"]
B2 --> R1
R1 --> B2
B2 --> C2
end

CIF["CHRONOLOGICAL INPUT FAILURE\nSystem-wide loss of chronological integrity under distributed execution pressure.\nEmergent condition, not a component failure"]

class CIF condition

subgraph RECON["4. RECONCILIATION LAYER: Out-of-Band Mitigation"]
R["Reconciliation System\nLog-based correction of financial state\nPartial, post-facto mitigation"]
end

A -->|Intended order| B1
A -->|Distributed execution| B2
C2 -.->|Triggers CIF state| CIF
C2 --> R

style INTENT fill:#F1F5F9,stroke:#64748B
style IDEAL fill:#FAFAFA,stroke:#94A3B8
style REAL fill:#FAFAFA,stroke:#94A3B8
style RECON fill:#F0FDF4,stroke:#16A34A

The critical insight this diagram reveals is the gap between the Deterministic Model column and the Distributed Execution column.

Engineers design for the left side. Production systems live on the right side. CIF is what emerges in the space between them.

CIF Classification — The Four Level Taxonomy

Not all CIF is equal. The framework classifies CIF into four levels of severity based on observable business impact.

Level 1 — Sync Lag
The core banking application and user application fall out of sync. Users see stale balances. Support tickets spike. Engineering investigates and finds nothing conclusively wrong. The system is technically correct but financially misleading.
Business impact: High support operational expenditure. Low immediate financial risk.

Level 2 — Event Replay
Duplicate credits appear during network retries. The same transaction processes twice. Finance teams notice discrepancies at month end. Direct revenue leakage begins here.
Business impact: Measurable financial leakage. Reconciliation backlog grows.

Level 3 — Non-Monotonicity
Sequence IDs invert. Transaction ordering breaks. Fraud detection models start training on out-of-order data. Model accuracy degrades silently. Engineering bandwidth gets consumed manually patching database rows.
Business impact: Engineering burn. ML model drift. Compliance exposure begins.

Level 4 — Causality Collapse
Total loss of ledger truth. Validation no longer reliably precedes commit. Regulators ask for an audit trail that cannot be produced. The reconciliation system becomes the primary source of truth, which it was never designed to be.Business impact: Regulatory failure. Board level exposure. Potential licence risk.

The Metrics Framework — Making CIF Measurable

The CIF framework introduces three metrics that translate distributed systems behavior into business risk language.

5.1 Causality Violation Rate
The CVR measures the frequency at which commit events are processed before their corresponding validation events complete.

CVR = (Number of commits processed before validation) / (Total transactions processed) x 100

A CVR of 0% means every commit followed a completed validation. Every transaction is provably ordered. Your ledger is truthful.

A CVR above 0% means your ledger contains transactions that were committed before they were validated. The higher the CVR the more your ledger diverges from financial truth under load.

Our baseline simulation produced a CVR of 8.3% under realistic conditions. This means 1 in every 12 transactions in an unprotected async pipeline commits before its validation completes.

5.2 Reconciliation Debt Index
The RDI translates your CVR into operational cost. It answers the question every CFO actually cares about: what is this costing us?
RDI = (CVR x Daily Transaction Volume x Escalation Rate x Resolution Time x Engineer Hourly Cost)
Using conservative assumptions:
Daily transaction volume: 1,000,000
CVR: 8.3% = 83,000 violations per day
Escalation rate: 1% requiring manual intervention = 830 incidents
Resolution time: 45 minutes per incident
Senior engineer hourly cost: $80
RDI = 830 x 0.75 hours x $80 = $49,800 per day

At scale, an unaddressed CIF problem costs approximately $49,800 per day in engineering overhead alone. This does not include financial leakage from duplicate credits, regulatory penalty risk, or the cost of model retraining caused by corrupted training data.

5.3 Ledger Integrity Score
The LIS is a composite metric that combines CVR and RDI into a single number representing the overall integrity health of your payment pipeline.

LIS = 100 - (CVR x 10) - (RDI normalised to a 0-10 scale)

A perfect LIS of 100 means zero causality violations and zero reconciliation debt. Your ledger is provably truthful.

A declining LIS is an early warning signal that your pipeline is accumulating silent integrity debt that will eventually surface as a reconciliation crisis, a compliance failure, or both.

Simulation Design

The simulation models a minimal but realistic async payment pipeline with two event types per transaction.

VALIDATE — representing an upstream webhook such as a fraud check or KYC verification.
COMMIT — representing the ledger write.
Both events are scheduled independently, reflecting real async pipeline behavior. Network delay is injected probabilistically on the validation path only, representing realistic jitter and retry behavior.

Simulation parameters:

Parameter. Value.

Total transactions. 5000

Validation delay probability. 8%

Maximum delay units. 5.0

Workers. 4

Partitions. 4

Maximum retries (safeguarded) 3

Initial backoff (safeguarded) 0.2

Baseline simulation code:
import random
import heapq

NUM_TRANSACTIONS = 5000
VALIDATION_DELAY_PROB = 0.08
MAX_DELAY = 5.0

events = []
validated = set()
violations = 0

for tx_id in range(NUM_TRANSACTIONS):
base_time = tx_id
validation_time = base_time
if random.random() < VALIDATION_DELAY_PROB:
validation_time += random.uniform(1, MAX_DELAY)
commit_time = base_time + 0.5
events.append((validation_time, "VALIDATE", tx_id))
events.append((commit_time, "COMMIT", tx_id))

heapq.heapify(events)

while events:
time, event_type, tx_id = heapq.heappop(events)
if event_type == "VALIDATE":
validated.add(tx_id)
elif event_type == "COMMIT":
if tx_id not in validated:
violations += 1

print("Total Transactions:", NUM_TRANSACTIONS)
print("Total Violations:", violations)
print("Violation Rate:", round((violations / NUM_TRANSACTIONS) * 100, 4), "%")

Results

Baseline output — no safeguards:
Total Transactions: 5000
Total Violations: 416
Violation Rate: 8.32%

This result is consistent and reproducible across multiple runs. The violation rate does not fluctuate dramatically because the delay probability is stable. This is a measurable, stable signal not random noise.

Sensitivity observations:
Increasing the validation delay probability from 8% to 15% raises the CVR to approximately 14%. The relationship is roughly linear within the parameters tested.

Increasing concurrency without adding ordering safeguards amplifies the CVR non-linearly. At higher concurrency levels partition rebalancing introduces additional ordering violations beyond the baseline delay probability.

The Three Safeguards — Reducing CVR to Near Zero

With three safeguards implemented the simulation produces dramatically different results.

Safeguard 1 — Partition Aware Routing
All events for the same transaction always route to the same worker via consistent hashing. This eliminates cross-worker ordering violations.
assigned_worker = (tx_id % NUM_PARTITIONS) % NUM_WORKERS

Safeguard 2 — Exponential Backoff
When a commit arrives before validation completes, instead of recording a violation the system retries with exponential backoff, giving the validation time to complete.
if retry_count < MAX_RETRIES:
backoff = INITIAL_BACKOFF * (2 ** retry_count)
return (current_time + backoff, "COMMIT", tx_id, retry_count + 1)

Safeguard 3 — Idempotency Registry
Prevents duplicate commit processing during retry cycles, eliminating ghost balance creation.
if tx_id in idempotency_registry:
return None

Safeguarded output:
Total Transactions: 5000
Causality Violations: 0
Violation Rate: 0.0%

None of these safeguards reduce throughput. They change how edge cases are handled without touching the happy path.

Economic Impact

The business translation of these results is straightforward.

At one million daily transactions without safeguards a CVR of 8.3% produces 83,000 causality violations per day. With a conservative 1% escalation rate requiring manual engineering intervention, that is 830 incidents per day requiring 45 minutes each at $80 per hour.

Daily reconciliation overhead: $49,800
Monthly reconciliation overhead: $1,494,000
Annual reconciliation overhead: $17,928,000

These figures do not include financial leakage from duplicate credits, regulatory penalty exposure, the cost of retraining fraud models on corrupted data, or the opportunity cost of engineering bandwidth diverted from product development to data patching.

For any fintech processing more than 100,000 transactions daily, measuring your CVR before your next regulatory audit is not optional. It is urgent.

Mitigation Strategies

Beyond the three safeguards demonstrated in the simulation, production systems should also consider:

The outbox pattern ensures events are written to a transactional outbox before being published to the message queue, guaranteeing at-least-once delivery with idempotency controls.

Event sourcing treats every state change as an immutable event, making the full causal history of any transaction reconstructible at any point.

Monotonic event IDs assign strictly increasing identifiers to events at the source, making ordering violations detectable at the consumer layer before they reach the ledger.

Limitations

This simulation is a behavioral model not a production system. It does not simulate real Kafka brokers or consumer groups, actual database writes, network topology, production retry policies with jitter, fraud patterns, database failures, or human reconciliation heuristics.

Results reflect simulation parameters and should not be treated as production benchmarks. The 8.3% CVR is a baseline measurement under specific conditions, not a universal claim about all async payment systems.

The value of this simulation is not in the specific number it produces. It is in demonstrating that causality violations are measurable, reproducible, and quantifiable before they become a production crisis.

Future Work

Several extensions of this framework are worth pursuing:

A CIF benchmark standard that allows fintech engineering teams to compare their CVR against industry baselines across different architectural patterns.

A Fintech Integrity Index that tracks ledger integrity health across the Nigerian and African fintech ecosystem as a public benchmark.

Real world trace validation using anonymised production logs to validate simulation parameters against actual system behavior.

Cross system comparison measuring CVR differences between Kafka, RabbitMQ, and other message queue implementations under equivalent load conditions.

Conclusion

Chronological Input Failure is not a bug. It is not a configuration error. It is a measurable property of distributed financial systems that emerges under concurrency and retry pressure.

At low transaction volumes it is invisible. At scale it becomes expensive. At regulatory audit time it becomes existential.

The CIF framework gives engineering and risk teams three tools they did not previously have: a way to measure how often their pipeline breaks causality, a way to quantify what that costs, and a way to track whether interventions are working.

System success and financial truth are not the same thing. The gap between them is measurable. And what is measurable can be fixed.

Full simulation code:
https://github.com/oludeleoluwapelumi/cif-simulation

How Three Safeguards Reduced Causality Violations From 8.3% to Near Zero in an Async Payment Pipeline

oludeleoluwapelumi — Sun, 28 Jun 2026 23:48:05 +0000

A few weeks ago I published a simulation showing that causality violations in async payment pipelines occur at a rate of 8.3% under realistic retry conditions.

For context, a causality violation happens when a commit hits the ledger before its corresponding validation completes. The transaction shows success. The balance updates. Everything looks fine on the surface. But the state was never properly validated before it became permanent.

At one million daily transactions that is 83,000 unvalidated commits every day accumulating silently into reconciliation debt.

Several engineers asked the same question after reading that article.

What actually fixes it?

This article answers that question with code.

The Baseline Problem
The baseline simulation models a simple two event transaction lifecycle:

VALIDATE event, which represents an upstream webhook or fraud check, and COMMIT event, which represents the ledger write.
Both events are scheduled independently in an async pipeline. When the validation path experiences network delay the commit can arrive first. The ledger writes before validation completes.

Here is the baseline simulation output across 5,000 transactions with 8% network retry probability:

Total Transactions: 5000
Total Violations: 416
Violation Rate: 8.32%
Consistent across every run. Reproducible. Measurable.

Now here is how to fix it.

Safeguard 1: Partition Aware Routing
The first problem is that events for the same transaction can be processed by different workers. When VALIDATE and COMMIT hit different workers there is no shared state to enforce ordering.

The fix is consistent hashing.Every event for the same transaction always routes to the same worker.

assigned_worker = (tx_id % NUM_PARTITIONS) % NUM_WORKERS

This alone does not eliminate violations. A delayed VALIDATE event on the same worker can still arrive after COMMIT. But it removes cross worker ordering chaos and creates the foundation for the next two safeguards.

Safeguard 2: Exponential Backoff On The Commit Path

When a COMMIT event arrives before its VALIDATE event has been observed, instead of immediately recording a violation the system schedules a retry with exponential backoff.

def handle_commit(tx_id, current_time, retry_count):
if tx_id in idempotency_registry:
return None

if tx_id in validated_db:
idempotency_registry[tx_id] = "COMMITTED"
return None

if retry_count < MAX_RETRIES:
backoff = INITIAL_BACKOFF * (2 ** retry_count)
return (current_time + backoff, "COMMIT", tx_id, retry_count + 1)

causality_violations += 1
return None

This gives delayed VALIDATE events time to complete before the commit is finalized. Most violations are eliminated at this stage.

Safeguard 3: Idempotency Registry

When retries occur there is a risk of processing the same commit multiple times. The idempotency registry ensures each transaction commits exactly once regardless of how many retry attempts occur.

if tx_id in idempotency_registry:
return None

This prevents ghost balances caused by duplicate commit processing during retry cycles.

The Results
With all three safeguards active the simulation output changes dramatically:

Total Transactions: 5000
Causality Violations: 0
Violation Rate: 0.0%

The key insight is that none of these safeguards slow down the pipeline. They change the handling of edge cases without touching the happy path.
Throughput remains the same. Integrity is enforced.

The Business Translation
At one million daily transactions:

Without safeguards: 83,000 causality violations per day requiring manual review.

With safeguards: Near zero violations. Reconciliation overhead drops to negligible levels. Your audit trail becomes provably ordered.

For any fintech preparing for a regulatory audit or CBN 2027 compliance review the difference between these two states is the difference between a clean audit and an operational crisis.

Important Caveats
This simulation is a behavioral model not a production Kafka implementation. It does not simulate real Kafka brokers or consumer groups, actual database writes, network topology, or production retry policies with jitter.

It models causal ordering behavior only. Results reflect simulation parameters. Real production environments will have additional complexity.

The full simulation code for both baseline and safeguarded versions is open source at github.com/yakuburoseline1-gif/cif-simulation

What To Do With This
If your engineering team is running a high throughput async payment pipeline, three questions worth asking:

Are validation and commit events routed to the same worker for the same transaction?

Does your commit path retry when validation has not yet completed?
Do you have idempotency controls preventing duplicate commit processing on retries?

If the answer to any of these is no or we are not sure, your pipeline may be producing causality violations at a rate similar to the baseline simulation.

The simulation takes under a minute to run. You can benchmark your own retry probability against it and estimate your violation rate before investing in a full architectural review.

I research causality violations in async payment pipelines and their operational impact on fintech ledgers.

Full simulation code at https://github.com/oludeleoluwapelumi/cif-simulation

The Silent Ledger Leak: Measuring Causality Violations in Async Payment Pipelines

oludeleoluwapelumi — Tue, 23 Jun 2026 03:22:32 +0000

I spent the last few months trying to understand why reconciliation errors keep appearing in high-throughput pipelines. Here is what I found.
In the race to process millions of transactions daily, modern fintech ecosystems have achieved a genuine miracle of scale. But beneath the surface of that velocity lies a structural problem most engineering teams aren't measuring: causality violations in async event pipelines.

Most teams assume that if a transaction shows "Success" in the database, the job is done. At high concurrency levels, that assumption breaks quietly.
When "Eventual Consistency" Becomes "Eventual Loss"
In distributed systems, Kafka partitions and database shards experience micro-millisecond timing gaps. When a network retry delays a validation webhook, the downstream ledger can commit a wallet update before the validation that should have preceded it completes.
To the user, the app glitches. To the engineering team, it's a reconciliation ticket. To the CFO, it's untracked operational cost.

The Reconciliation Tax
I built a simulation modelling this exact failure mode across 5,000 concurrent transactions. With an 8% network retry probability, conservative for high-traffic payment rails, the causality violation rate was 8.3%.

At one million daily transactions, that's over 80,000 unvalidated commits every day requiring manual review.
The operational cost compounds across three dimensions: engineering hours spent patching database state, fraud model accuracy degrading on out-of-order training data, and audit trails that cannot demonstrate strict causal sequence to regulators.

The Fix
The solution is enforcing strict event ordering at the ingestion layer before state commits happen, not better monitoring after the fact.
When safeguards including partition-aware routing, exponential backoff, and idempotency controls were added to the same simulation, the violation rate dropped to 0%.

Full simulation code and methodology:
https://github.com/oludeleoluwapelumi/cif-simulation

Are you measuring your pipeline's causality violation rate?