Mitigating the Two Generals Problem in Django with Idempotence

#django #distributedsystems #backend #fintech

A Fintech Architecture Perspective

When building payment systems, you eventually face a terrifying question:

Did we charge the customer or not?

The worst part?

Sometimes, the system genuinely doesn't know.

This is not a bug. It's a distributed systems reality described by the Two Generals Problem: two parties can not guarantee agreement over an unreliable network.

In fintech, this becomes:

Merchant --> Backend --> Payment Provider
Charge succeds
Network drops
Backend never receives confirmation
Client retries

Now what?

If you retry blindly, you double charge.

If you don't retry, you might lose revenue.

This is where idempotency become a core architectural pattern.

Why this matters in Fintech

In international payment systems:

Mobile networks are unstable
Clients double-click
Workers crash mid-transaction

If your backend is not idempotent, you will:

Double charge customers
Create accounting inconsistencies
Trigger reconciliation nightmares
Lose merchant trust

Large providers like Stripe formalized idempotency because this problem is structural, not incidental.

The Goal: Deterministic Retries

We cannot guarantee exactly-once execution over HTTP.

What we can guarantee:

The same request key always produces the same result.

That's the architectural shift. Instead of solving uncertainty, we make retries safe.

Designing Idempotency in Django

This is not about adding a if exists check. This is about enforcing correctness at the database boundary.

1. Require an Idempotency Key

Idempotency-Key: 550e8400-e29b-41d4-a716-446655440000

This key represents one logical business operation.

2. Persist the Key with a Unique Constraint

# models.py

class IdempotencyRecord(models.Model):
    key = models.CharField(max_length=255, unique=True)
    request_hash = models.CharField(max_length=64)
    response_body = models.JSONField(null=True, blank=True)
    response_status = models.IntegerField(null=True, blank=True)
    created_at = models.DateTimeField(auto_now_add=True)

Why this matters:

unique=True protects against concurrency.
The database becomes the source of truth.
Race conditions move from application logic to atomic constraints.

This is architectural, not syntax.

3. Hash the Request Payload

import hashlib

def hash_request(request):
    return hashlib.sha256(request.body).hexdigest()

Why ?

If someone reuses the same key with a different payload, that's dangerous. We must detect:

Same key
Different business intent

And reject it.

The hash is generated on the server when Django receives the request.

request payload -> server-side hash computation 
                -> database storage 
                -> hash comparison on retries 
                -> deterministic decision (replay or reject)

4. Atomic Handling

from django.db import transaction, IntegrityError
from django.http import JsonResponse

@transaction.atomic 
def process_payment(request):
    key = request.headers.get("Idempotency-Key")
    if not key:
        return JsonResponse({"error": "Missing Idempotency-Key"}, status=400)

    request_hash = hash_request(request)

    try:
        record = IdempotencyRecord.objects.create(
            key=key,
            request_hash=request_hash
        )
        first_request = True
    except IntegrityError:
        record = IdempotencyRecord.objects.select_for_update().get(key=key)
        first_request = False

    if not first_request:
        if record.request_hash != request_hash:
            return JsonResponse(
            {"error": "Payload mismatch"},
            status=409
            )
        return JsonResponse(record.response_body, status=record.response_status)

    # External side effect (charge)
    charge = external_payment_call()

    response = {"payment_id": charge.id}

    record.response_body = response
    record.response_status = 200
    record.save()

    return JsonResponse(response, status=200)