TOCTOU in Payment Systems: When Validation Becomes Stale Before Commit

1. The formal problem: CWE-367

Time-of-check to time-of-use (TOCTOU) is a formally catalogued class of race condition, CWE-367, first documented in 2006 and still actively relevant two decades later. The pattern: a program evaluates a precondition, then acts on a resource whose state it assumes is unchanged, without any atomicity guarantee between the two operations.

The canonical C example:

// VULNERABLE: classic TOCTOU, access() then open()
void read_file(const char *filename) {
if (access(filename, R_OK) == 0) {
// RACE WINDOW: the filesystem state between these two
// calls is not guaranteed to be unchanged
FILE *fp = fopen(filename, "r");
// ...
}
}

access() and fopen() are two independent system calls. Nothing forces the kernel to guarantee the file referenced at access() time is the same file referenced at fopen() time. An attacker with write access to the containing directory can swap the target for a symlink to /etc/shadow inside that window, and the check will have been technically correct at the moment it ran.

Python has the identical pattern, and it's common enough that Semgrep ships a default rule for it:

FLAGGED by static analysis: TOCTOU filesystem race

if os.path.exists(path):
# RACE WINDOW
open(path, ...)

The fix in both cases is the same: collapse check and use into a single atomic operation, O_CREAT | O_EXCL for atomic file creation, fstat() on an already-open file descriptor instead of stat() on a path, or a held lock (flock(), fcntl(F_SETLK)) spanning both operations. The general principle: never trust a decision made against a resource you no longer have exclusive reference to.

This isn't a historical curiosity. Recent, high-severity instances include CVE-2019-5736, a TOCTOU race in runc that allowed a malicious container to overwrite the host runc binary, achieving container escape, and CVE-2016-9806, a race condition in Linux kernel netlink handling that enabled privilege escalation. Both post-date CWE-367's formal definition by a decade, in codebases with far more scrutiny than most application code ever receives.

1. Sequencing the race The following sequence diagram makes the actual failure mode explicit, timing, not logic, is the vulnerability:

sequenceDiagram
    participant P as Process
    participant FS as Filesystem
    participant A as Attacker

    P->>FS: access(path, R_OK)
    FS-->>P: OK (check passes)
    Note over A: Race window opens
    A->>FS: unlink(path); symlink(/etc/shadow, path)
    Note over A: Race window closes
    P->>FS: fopen(path, "r")
    FS-->>P: opens /etc/shadow, not the original file

The check and the use are both individually correct. The system's understanding of "what path refers to" is what's wrong, and it's wrong specifically because time passed between the two operations, and something else had write access during that window.

1. A formal distributed systems lens: commitment ordering Set the security literature aside. A parallel, independent body of work in distributed database theory has been formalizing a related concern since 1990, under the name commitment ordering (CO), introduced by Yoav Raz in "The Principle of Commitment Ordering, or Guaranteeing Serializability in a Heterogeneous Environment of Multiple Autonomous Resource Managers Using Atomic Commitment," VLDB 1992.

The formal property: in a CO-compliant schedule, the chronological order of transactions' commit events must be compatible with the precedence order of their conflicting operations. If transaction A's write conflicts with transaction B's write, the order in which A and B actually commit has to match the order in which their conflicting operations actually executed. It is not sufficient for each transaction to be individually correct in isolation. The commit sequence itself has to be honest about what really happened first.

To be precise about the relationship: this piece is not claiming that the specific failure CIF describes has been shown to satisfy the formal conditions of a CO violation, that would require a level of formal proof this piece doesn't attempt. What's fair to say is narrower and still useful: commitment ordering provides a formal, decades-old distributed-systems framework for reasoning about one dimension of the ordering problem CIF is concerned with, the question of whether commit order tracks real precedence, not just per-service execution order. CO was developed specifically to solve global serializability across autonomous resource managers, systems with no shared clock, no shared concurrency control, no assumption of coordination, which is structurally close to a modern payment pipeline: a validation service, a ledger, a fraud engine, often owned by different teams, running on different infrastructure.

The practical mechanisms CO analysis identifies for enforcing this, optimistic concurrency control and strict locking protocols that hold resources until commit, are the same categories showing up independently in the fixes discussed next.

1. Three engineers, three vocabularies, one underlying discipline

In three separate, unconnected conversations over the last couple of weeks, three engineers independently described related fixes for what they each thought was their own distinct problem. None of the code below is copied from them, it's my own implementation of the pattern each described, credited to the concept, not claimed as their literal source. These are not literally identical mechanisms, they're different implementations of the same underlying discipline, worth being precise about that distinction.

Alok Ranjan Daftuar named this "the TOCTOU family" directly, and proposed a conditional write, a commit that re-checks its precondition atomically, as part of the same statement that performs the write:

-- Conditional write: the WHERE clause re-verifies the precondition
-- atomically, as part of the same statement that performs the commit
UPDATE accounts
SET balance = balance - :amount,
version = version + 1
WHERE account_id = :account_id
AND version = :expected_version
AND validation_status = 'CONFIRMED';

-- If validation hasn't completed (validation_status != 'CONFIRMED'),
-- or another writer has already advanced the version, this UPDATE
-- affects zero rows. The application checks the row count and treats
-- zero as a rejected commit, not a silent no-op.

Emmanuel Valverde Ramos, writing about event sourcing, described optimistic concurrency via an expected revision check before appending:

def append_event(store, stream_id, expected_revision, event):
current_revision = store.get_stream_revision(stream_id)
if current_revision != expected_revision:
# Something committed to this stream since we last read it.
# Reload and let the caller re-decide, don't append blindly.
raise ConcurrencyConflict(stream_id, expected_revision, current_revision)
store.append(stream_id, event, expected_revision)

Mayckon Giovani, discussing multi-step transaction orchestration, warned against implicit sequencing, code order isn't guaranteed to be real-world order, and argued each step must revalidate its own preconditions at execution time:

async def execute_step(step, context):
# Re-check preconditions right here, at execution time,
# not once at the start of the saga and never again
if not await step.preconditions_still_valid(context):
return StepResult.ABORTED_STALE_PRECONDITION
return await step.execute(context)

The common abstraction:

flowchart LR
    A[Earlier check<br/>passes] -.->|time passes,<br/>state may change| B{Re-verify at<br/>the moment of commit}
    B -->|still valid| C[Commit proceeds]
    B -->|no longer valid| D[Commit refused,<br/>not silently allowed]

Three independent engineers, working on unrelated problems (SQL writes, event sourcing, saga orchestration), converged on the same defensive principle: don't rely indefinitely on a previously observed state. That convergence is the interesting evidence here, not a claim that any of them discovered or endorsed CIF specifically. They didn't. What it establishes is that this discipline recurs independently across distributed systems work, which is a meaningfully strong signal on its own.

1. Where CIF found this, and what testing it looked like Everything above establishes that this problem is old, formally studied in a related form, and independently rediscovered as a defensive pattern. None of that proves it happens in ordinary financial systems, not attacked, not adversarial, just under normal async load. Here's what testing that looked like, including its real limitations.

The mechanism, demonstrated with real concurrency. The following uses Python's asyncio event loop to run genuine concurrent validation and commit tasks, real timing-based races through the scheduler, not arithmetic timestamp comparison. It's important to be precise about what this experiment actually is: PaySim provides the transaction records as input content; it does not provide evidence that these specific transactions experienced the race condition being modeled. PaySim contains no validation or commit timestamps of its own. The concurrency behavior, the jitter, the timing, is injected by this experiment. This is real financial transaction data used as input to a synthetic concurrent execution model, not a claim that PaySim itself recorded a race.

async def validate(tx_id, ledger, jittered):
delay = random.uniform(0.02, 0.05) if jittered else random.uniform(0.0, 0.004)
await asyncio.sleep(delay)
ledger.validated.add(tx_id)

async def commit(tx_id, ledger, safeguard):
await asyncio.sleep(0.01) # fixed processing delay
if safeguard and tx_id not in ledger.validated:
for _ in range(3):
await asyncio.sleep(0.015)
if tx_id in ledger.validated:
ledger.recovered_after_wait += 1
break
else:
ledger.rejected_by_safeguard += 1
return "REJECTED_BY_SAFEGUARD"
if tx_id not in ledger.validated:
ledger.violations += 1
ledger.committed[tx_id] = True

Worth being precise about what this safeguard actually demonstrates: execution-time revalidation with bounded retry, not a fully atomic conditional commit. There's a real window in this code between checking tx_id in ledger.validated and setting ledger.committed[tx_id] = True; in a real shared database under real concurrency, state could still change inside that window. The SQL example in section 4 is the stronger, production-grade pattern, precondition and write in the same atomic statement. This demo shows a cleaner but weaker safeguard by design, useful for demonstrating the mechanism, not a claim of production-grade atomicity.

Run against 2,000 real PaySim rows, two independent runs, different random seeds, different data samples:

Run	Baseline violations	Safeguarded violations
1 (seed 42)	150 / 2,000 (7.50%)	0
2 (seed 123)	172 / 2,000 (8.60%)	0

Both baseline rates land close to the 8% injected jitter probability, expected, and both safeguarded runs hit zero, the circuit breaker held.

That's the favorable case. It's honest to also show the safeguard's limit, not just its success. Pushing the jitter delay past the retry window's maximum wait produces this instead:

Condition	Result
Baseline (no safeguard)	150 / 2,000 violations (7.5%)
Safeguarded, severe jitter	0 blind violations, 100 recovered within retry window, 50 rejected by safeguard

The safeguard never once let an invalid commit through. But it also didn't magically fix everything, when validation genuinely couldn't complete in time, it correctly refused the transaction rather than guessing. That's the actual tradeoff a conditional write or optimistic concurrency check buys: not "no failures," but "no silent failures." A rejected transaction is visible and recoverable. A blindly-committed one isn't.

sequenceDiagram
    participant G as Async gateway
    participant V as Validation task
    participant C as Commit task (with safeguard)
    participant L as Ledger

    G->>V: dispatch validate()
    G->>C: dispatch commit()
    Note over V: jitter delay (KYC/fraud check)
    C->>C: wait fixed processing delay
    C->>L: check: has V completed?
    alt validated in time
        C->>L: commit, record success
    else validation still pending
        C->>C: bounded retry (3x)
        alt caught up during retry
            C->>L: commit, record success
        else still not validated
            C->>L: REJECTED_BY_SAFEGUARD
        end
    end

The reconciliation check against real data. Separately, a static reconciliation check against ~77,000 real PaySim rows tested a related question: does the recorded balance change match the transaction that supposedly caused it? The first pass returned 75.92% flagged, wrong, and investigating why was itself the useful part of the exercise: two dataset-specific artifacts (untracked merchant balances, balances clipped to zero rather than negative) and one direction bug in the check itself (CASH_IN adds to a balance; every other type subtracts). Corrected, the result was 26 candidate anomalies, internally inconsistent records under the corrected reconciliation model, out of roughly 35,000 checkable transactions, 0.07%, mostly unflagged by any existing error or fraud indicator. Full methodology: PAYSIM_RECONCILIATION_CHECK.md. These are candidate anomalies under the model, not confirmed financial failures; the scanner's own documentation says as much.

The independent bug-history check. A taxonomy mapping against Apache Fineract's public JIRA history, an open-source core banking platform, found several tickets in related territory. FINERACT-1744 is worth being precise about: it is not itself a documented TOCTOU bug in the check/use sense above. It demonstrates a closely related failure mode, duplicate execution caused by retries, an operation running twice after an earlier execution had already completed, and the Fineract team responded by building dedicated, system-wide idempotency infrastructure (idempotency keys, command status tracking) rather than patching the specific incident. Full mapping, including one ticket explicitly ruled out on closer reading rather than forced to fit: FINERACT_TAXONOMY_MAPPING.md.

1. What's actually established, and what isn't

To state this precisely, evidence gathered so far falls into distinct categories, and they answer different questions:

Evidence	Question it answers
Asyncio experiment	Can this race mechanism occur at all? Yes, reproducibly, across independent runs.
PaySim reconciliation	Can real transaction records be examined for related integrity inconsistencies? Yes, a small, real signal, honestly caveated.
Fineract history	Do real financial systems have documented failures in closely related territory, retries, duplication, idempotency? Yes.
Production validation	Does this specific pattern occur in a currently operating payment company's production system? Unknown.

That last row is the honest, current status of this work: mechanism demonstrated, historical precedent established in related failure modes, public-data testing performed, production validation still outstanding. Not "validated." Not "discovered." A specific, testable hypothesis with three independent forms of supporting evidence and one still-missing form, a production engineer able to say "yes, we've seen this."

The code for all of the above, the asyncio demo, the reconciliation scanner, the Fineract taxonomy mapping, is public: https://github.com/oludeleoluwapelumi/cif-simulation

If you run a payment system and any of this maps onto how it's built, there's a free, offline scanner (/scanner) that runs entirely on your own machine, no data has to leave it, to check your own logs against the reconciliation pattern above. Genuinely interested in what it finds, including if the answer is nothing.