DEV Community: Hilda Enyioko

Two threads, one card, one invoice: hunting a TOCTOU bug in a payments codebase

Hilda Enyioko — Mon, 03 Aug 2026 08:27:20 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

TheeInsurance is a headless, API-first insurance platform for Nigeria. Insurance providers and distributors use it to manage plans, subscriptions, KYC, and payments through a multi-tenant Django REST backend. It integrates Interswitch Quickteller Pay and Nomba, including tokenized card storage for automated policy renewals.

Bug Fix or Performance Improvement

The bug lives in charge_policy_renewal().
This function is responsible for auto-charging a customer's stored card when their policy is due for renewal. Since renewals can be triggered by the scheduler, an n8n renewal call, or a dunning retry, concurrent execution can legitimately happen under production load or after retry timing overlaps.

To prevent double-charging, the function checked for an in-flight renewal before creating a new one:

in_flight = Transaction.objects.filter(
    subscription=sub,
    payment_type=Transaction.PAYMENT_TYPE.RENEWAL,
    payment_status=Transaction.PAYMENT_STATUS.PENDING,
    gateway=Transaction.GATEWAY.NOMBA,
).exists()

if in_flight:
    raise PaymentError("A renewal charge is already in progress.")

with db_transaction.atomic():
    txn = Transaction.objects.create(...)

This looks like a reasonable guard. But the .exists() check and the .create() are two separate operations, not covered by the same lock. If two callers hit this function within the same tiny window, both can evaluate in_flight = False before either has committed its new PENDING Transaction. Both then proceed to call Nomba's tokenized charge endpoint. Same card, same policy, two charges.

This is a textbook TOCTOU (time-of-check to time-of-use) race condition. In a payments codebase like this one, it's not an edge case worth shrugging off. It's a customer getting charged twice for the same policy, with the operational cost of a refund and a support ticket on top.

Code

Full PR: [https://github.com/Hilda-Enyioko/theeinsurance_infrastructure/pull/8]

Reproduction test: payments/tests/test_renewal_race_condition.py fires two concurrent calls to charge_policy_renewal() for the same subscription, using a threading.Barrier to force the interleaving deterministically (relying on raw OS thread timing would make a race-condition test flaky and unconvincing).

Before the fix:

AssertionError: Expected exactly 1 successful renewal charge, got 2. Blocked: 0.
Race condition allowed duplicate billing.

10/10 manual runs reproduced the duplicate charge.

BEFORE FIX TEST RESULT: race_condition_results.txt

After the fix:

successes: 1, blocked: 1, errors: 0
renewal_txn_count: 1

0/10 runs produced a duplicate charge.

AFTER FIX TEST RESULT: sentry_race_test_output.txt

My Improvements

I fixed this at two layers, because I wanted both the correct fix and a guaranteed fix:

1. Close the race at the application level. The subscription row is now locked with select_for_update() at the very top of the function. Since every renewal targets a single subscription, locking that row naturally serializes renewal attempts for the same policy without reducing concurrency across unrelated subscriptions. The entire check-then-act sequence, status checks, the in-flight query, and the Transaction.objects.create(), happens inside that one atomic() block:

with db_transaction.atomic():
    sub = PolicySubscription.objects.select_for_update().get(id=subscription_id)
    # ... validation ...
    in_flight = Transaction.objects.filter(...).exists()
    if in_flight:
        raise PaymentError(...)
    txn = Transaction.objects.create(...)
# lock released here — the Nomba HTTP call happens outside the transaction

A second concurrent call now blocks on the row lock until the first transaction commits, then correctly sees the in-flight renewal and backs off. I deliberately kept the outbound Nomba call outside the atomic block. This is because holding a row lock during a 30-second-timeout network call would create its own problems (blocking legitimate concurrent reads on that subscription for the duration of a slow gateway call).

2. Add a database-level guarantee as a second line of defense. Application logic can have bugs, or a future code path might bypass this function entirely. So I added a partial unique constraint:

migrations.AddConstraint(
    model_name='transaction',
    constraint=models.UniqueConstraint(
        fields=['subscription', 'payment_type', 'gateway'],
        condition=models.Q(payment_status='pending'),
        name='unique_pending_renewal_per_subscription',
    ),
)

Even if the application-level lock is ever removed or bypassed, Postgres itself now refuses to store a second concurrent PENDING renewal row for the same subscription. I think of the row lock as the "correct" fix and the constraint as the "can never regress" fix.

The trickiest part of this whole exercise honestly was writing a test that could reliably prove the race existed in the first place, given that race conditions are timing-dependent by nature. Using a threading.Barrier patched into the .filter() call let me force both threads to reach the vulnerable window at the same instant, every time, instead of hoping for an unlucky interleaving.

Best Use of Sentry

I instrumented charge_policy_renewal() with a Sentry transaction and spans around each meaningful step so the fix is observable in production behavior:

with sentry_sdk.start_transaction(op="renewal", name="charge_policy_renewal") as sentry_txn:
    with db_transaction.atomic():
        with sentry_sdk.start_span(op="db.lock", description="select_for_update subscription"):
            sub = PolicySubscription.objects.select_for_update().get(id=subscription_id)

        in_flight = Transaction.objects.filter(...).exists()
        if in_flight:
            sentry_sdk.set_tag("renewal.duplicate_blocked", True)
            sentry_sdk.capture_message(
                f"Blocked duplicate renewal charge attempt for subscription {subscription_id}",
                level="warning",
            )
            raise PaymentError(...)

Running the concurrency test against this instrumented version, I could see:

A Performance trace for charge_policy_renewal showing the db.lock span. The second thread's wait time on the row lock is directly visible in the waterfall.
A warning-level Issue ("Blocked duplicate renewal charge attempt...") firing exactly once, from the thread that correctly got blocked.

I set send_default_pii=False given this codebase handles KYC and payment data.
I'd dial traces_sample_rate down from 1.0 before this runs against real production traffic. 100% tracing on every payment call is unnecessary overhead at scale, 1.0 was just useful for capturing this demo cleanly.

Best Use of Google AI

After identifying and fixing the race condition manually, I ran the original failing test's traceback and the pre-fix function source through Gemini as an independent sanity check on my root-cause reasoning:

import google.generativeai as genai

genai.configure(api_key=settings.GOOGLE_AI_API_KEY)
model = genai.GenerativeModel("gemini-2.0-flash")

response = model.generate_content(f"""
Given this test failure and the relevant function source, identify the
root cause and suggest a fix:

FAILURE:
{failure_traceback}

FUNCTION:
{charge_policy_renewal_source}
""")

Gemini independently converged on the same diagnosis: a TOCTOU gap between the .exists() check and the Transaction.create() call, with a select_for_update()-based lock as the standard fix. I ran this as a standalone diagnostic script not as part of the application itself. I took this route because this is a payments code path. I didn't want to add an external AI dependency or extra latency to a real charge flow.

Thanks for organizing the challenge. It was a great excuse to revisit an unfinished project. And, it ended up uncovering a race condition that could have resulted in real customers being charged twice. Those are exactly the kinds of bugs worth fixing.

"It's Not Working" Is Not an Error

Hilda Enyioko — Sat, 01 Aug 2026 21:31:23 +0000

Three weeks ago, the frontend developer on a team I manage reported an error she was facing:

"It's not working!"

Two days ago, another developer at work reported an error:

"500 Internal Server Error..."

No screenshot. No error message. No endpoint. No timestamp.
Just four words can mean anything, like a user forgetting to click "Save."

I've worked across multiple projects as a backend engineer, and I currently lead the build team behind FUTO Aid, our flagship student fundraising platform at IDC-FUTO. Leading a cross-functional team means I spend almost every day at the intersection of frontend and backend development.

One recurring friction point I've noticed is that frontend developers, who understand how software systems work, often report bugs the way end users do.

It isn't necessarily because they don't know better. More often, no one has shown them what good error reporting looks like or why it matters.

This article isn't a rant. It's the guide I wish existed the first time I sat down with a teammate and said, "I can't fix what I can't see."

Why Does Proper Error Reporting Matter?

When a bug is reported vaguely, the backend engineer becomes a detective before they can become a problem-solver.

For every vague reporting, we have the same extensive loop:

"What were you doing when it happened?"
"Can you send a screenshot?"
"Which page/button/action?"
"Is this still happening?"

Before the real debugging starts, there's already an entire slack thread, or a 10-min (could be longer) call trying to reproduce the issue reported. Multiply that across a sprint, across a team, across a project timeline. It adds up to real, avoidable delay.

Good error reporting is a force multiplier. It's often the difference between a bug fixed in 15 minutes and one that takes up half a sprint.

What Makes a Developer's Error Report Different From a User's?

I like to tell engineers on my team that while end users describe symptoms, developers should report evidence.

An end user is allowed to say "it's not working." They don't have access to dev tools, they don't know what an endpoint is, and they shouldn't have to.

A developer reporting a bug, especially to another developer, has no such excuse. Precision and clarity is the key differentiator.

What makes an Error Report Developer-Level Precise and Clear?

1. Copy the Exact Error Message:

This isn't the UI-wrapped error. This is the literal text from the literal error from the browser console, copied — not paraphrased. Paraphrasing an error message makes it lose its 'stew'. It loses the specific detail (error codes, stack traces, variable names) that often points straight to the cause.

2. Get the Failing Endpoint:

Which API call actually failed? This is visible in the Network tab. Rather than guessing across an entire codebase, it immediately tells the backend engineer which controller, service, or route to look at.

3. Note The Exact Time or Time Range:

This is a crucial part of reporting to help the backend engineer easily trace the logs on server-side produced by the failing endpoint. It is important to note that, "It happened this morning" is not a timestamp.

4. Copy the Request and Response Details:

The status code (400? 401? 500?), the request payload, and the response body are all visible in the Network tab. A good 70% of the time, they reveal why something failed, not just that it failed.

5. Provide Steps to Reproduce:

What sequence of actions led to the error? Can it be triggered consistently, or was it a one-off?

6. Share What's Already Been Tried:

This one matters more between developers than anywhere else. If you're a backend developer reporting an issue to another backend developer, saying "I tried restarting the queue worker and clearing the Redis cache, still fails" saves your teammate from repeating dead ends.

7. Provide Environment Context:

Local, staging, or production? Which branch or deployment? A bug that only happens in staging because of a stale environment variable looks very different from one that's broken everywhere.

A Brief Walkthrough: Using the Network Tab to Report Errors Properly

If there's one browser tool every frontend developer should be fluent in before reporting a bug to a backend engineer, it's the Network tab.

Here's a simple walkthrough:

Open dev tools (F12 or right-click → Inspect) and go to the Network tab.
Reproduce the issue — perform the action that triggers the bug while the Network tab is recording.
Find the failing request. Failed requests are usually easy to spot — they'll show a red status or a 4xx/5xx status code in the Status column.
Click on that request. This opens a detail panel with several useful tabs:
- Headers — shows the endpoint URL, request method (GET, POST, etc.), and status code.
- Payload/Request — shows exactly what data was sent to the server.
- Response — shows exactly what the server sent back, often including a specific error message from the backend itself.
- Timing — shows how long the request took, useful for diagnosing timeouts.
Copy, don't summarize. Copy the request URL, the response body, and the status code directly into your bug report. A screenshot of this panel works too — as long as all four pieces (URL, method, status, response) are visible.
Cross-check the Console tab for any related JavaScript errors that occurred alongside the failed request.

If you've never used the Network tab while reporting a bug, this habit alone will improve the quality of your reports overnight.

Good bug report

POST /api/user/avatar-upload returned 413 Payload Too Large at 2:14 PM.
The request payload exceeded the upload size limit.

This single sentence tells the backend engineer exactly where to look, cutting the back-and-forth down to zero.

Conclusion

None of this is complicated. It's not asking frontend developers to understand backend architecture, or expecting anyone to debug someone else's code. It's just asking that reports come from a developer's vantage point.

Small habits like this compound. They reduce the back-and-forth between developers and development units, cut down the time spent debugging and deciphering vague reports, and, maybe most importantly, they build a culture of precision within a team. On teams like IDC-FUTO Build Team, where everyone's time is already stretched thin, that precision isn't a nice-to-have. It's what keeps a project moving.

I'd honestly love to see developers do better when communicating issues.