Two threads, one card, one invoice: hunting a TOCTOU bug in a payments codebase

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

TheeInsurance is a headless, API-first insurance platform for Nigeria. Insurance providers and distributors use it to manage plans, subscriptions, KYC, and payments through a multi-tenant Django REST backend. It integrates Interswitch Quickteller Pay and Nomba, including tokenized card storage for automated policy renewals.

Bug Fix or Performance Improvement

The bug lives in charge_policy_renewal().
This function is responsible for auto-charging a customer's stored card when their policy is due for renewal. Since renewals can be triggered by the scheduler, an n8n renewal call, or a dunning retry, concurrent execution can legitimately happen under production load or after retry timing overlaps.

To prevent double-charging, the function checked for an in-flight renewal before creating a new one:

in_flight = Transaction.objects.filter(
    subscription=sub,
    payment_type=Transaction.PAYMENT_TYPE.RENEWAL,
    payment_status=Transaction.PAYMENT_STATUS.PENDING,
    gateway=Transaction.GATEWAY.NOMBA,
).exists()

if in_flight:
    raise PaymentError("A renewal charge is already in progress.")

with db_transaction.atomic():
    txn = Transaction.objects.create(...)

This looks like a reasonable guard. But the .exists() check and the .create() are two separate operations, not covered by the same lock. If two callers hit this function within the same tiny window, both can evaluate in_flight = False before either has committed its new PENDING Transaction. Both then proceed to call Nomba's tokenized charge endpoint. Same card, same policy, two charges.

This is a textbook TOCTOU (time-of-check to time-of-use) race condition. In a payments codebase like this one, it's not an edge case worth shrugging off. It's a customer getting charged twice for the same policy, with the operational cost of a refund and a support ticket on top.

Code

Full PR: [https://github.com/Hilda-Enyioko/theeinsurance_infrastructure/pull/8]

Reproduction test: payments/tests/test_renewal_race_condition.py fires two concurrent calls to charge_policy_renewal() for the same subscription, using a threading.Barrier to force the interleaving deterministically (relying on raw OS thread timing would make a race-condition test flaky and unconvincing).

Before the fix:

AssertionError: Expected exactly 1 successful renewal charge, got 2. Blocked: 0.
Race condition allowed duplicate billing.

10/10 manual runs reproduced the duplicate charge.

BEFORE FIX TEST RESULT: race_condition_results.txt

After the fix:

successes: 1, blocked: 1, errors: 0
renewal_txn_count: 1

0/10 runs produced a duplicate charge.

AFTER FIX TEST RESULT: sentry_race_test_output.txt

My Improvements

I fixed this at two layers, because I wanted both the correct fix and a guaranteed fix:

1. Close the race at the application level. The subscription row is now locked with select_for_update() at the very top of the function. Since every renewal targets a single subscription, locking that row naturally serializes renewal attempts for the same policy without reducing concurrency across unrelated subscriptions. The entire check-then-act sequence, status checks, the in-flight query, and the Transaction.objects.create(), happens inside that one atomic() block:

with db_transaction.atomic():
    sub = PolicySubscription.objects.select_for_update().get(id=subscription_id)
    # ... validation ...
    in_flight = Transaction.objects.filter(...).exists()
    if in_flight:
        raise PaymentError(...)
    txn = Transaction.objects.create(...)
# lock released here — the Nomba HTTP call happens outside the transaction

A second concurrent call now blocks on the row lock until the first transaction commits, then correctly sees the in-flight renewal and backs off. I deliberately kept the outbound Nomba call outside the atomic block. This is because holding a row lock during a 30-second-timeout network call would create its own problems (blocking legitimate concurrent reads on that subscription for the duration of a slow gateway call).

2. Add a database-level guarantee as a second line of defense. Application logic can have bugs, or a future code path might bypass this function entirely. So I added a partial unique constraint:

migrations.AddConstraint(
    model_name='transaction',
    constraint=models.UniqueConstraint(
        fields=['subscription', 'payment_type', 'gateway'],
        condition=models.Q(payment_status='pending'),
        name='unique_pending_renewal_per_subscription',
    ),
)

Even if the application-level lock is ever removed or bypassed, Postgres itself now refuses to store a second concurrent PENDING renewal row for the same subscription. I think of the row lock as the "correct" fix and the constraint as the "can never regress" fix.

The trickiest part of this whole exercise honestly was writing a test that could reliably prove the race existed in the first place, given that race conditions are timing-dependent by nature. Using a threading.Barrier patched into the .filter() call let me force both threads to reach the vulnerable window at the same instant, every time, instead of hoping for an unlucky interleaving.

Best Use of Sentry

I instrumented charge_policy_renewal() with a Sentry transaction and spans around each meaningful step so the fix is observable in production behavior:

with sentry_sdk.start_transaction(op="renewal", name="charge_policy_renewal") as sentry_txn:
    with db_transaction.atomic():
        with sentry_sdk.start_span(op="db.lock", description="select_for_update subscription"):
            sub = PolicySubscription.objects.select_for_update().get(id=subscription_id)

        in_flight = Transaction.objects.filter(...).exists()
        if in_flight:
            sentry_sdk.set_tag("renewal.duplicate_blocked", True)
            sentry_sdk.capture_message(
                f"Blocked duplicate renewal charge attempt for subscription {subscription_id}",
                level="warning",
            )
            raise PaymentError(...)

Running the concurrency test against this instrumented version, I could see:

• A Performance trace for charge_policy_renewal showing the db.lock span. The second thread's wait time on the row lock is directly visible in the waterfall.
• A warning-level Issue ("Blocked duplicate renewal charge attempt...") firing exactly once, from the thread that correctly got blocked.

I set send_default_pii=False given this codebase handles KYC and payment data.
I'd dial traces_sample_rate down from 1.0 before this runs against real production traffic. 100% tracing on every payment call is unnecessary overhead at scale, 1.0 was just useful for capturing this demo cleanly.

Best Use of Google AI

After identifying and fixing the race condition manually, I ran the original failing test's traceback and the pre-fix function source through Gemini as an independent sanity check on my root-cause reasoning:

import google.generativeai as genai

genai.configure(api_key=settings.GOOGLE_AI_API_KEY)
model = genai.GenerativeModel("gemini-2.0-flash")

response = model.generate_content(f"""
Given this test failure and the relevant function source, identify the
root cause and suggest a fix:

FAILURE:
{failure_traceback}

FUNCTION:
{charge_policy_renewal_source}
""")

Gemini independently converged on the same diagnosis: a TOCTOU gap between the .exists() check and the Transaction.create() call, with a select_for_update()-based lock as the standard fix. I ran this as a standalone diagnostic script not as part of the application itself. I took this route because this is a payments code path. I didn't want to add an external AI dependency or extra latency to a real charge flow.

---

Thanks for organizing the challenge. It was a great excuse to revisit an unfinished project. And, it ended up uncovering a race condition that could have resulted in real customers being charged twice. Those are exactly the kinds of bugs worth fixing.