DEV Community

Paul Owuor
Paul Owuor

Posted on

Designing Resilient Payment Microservices: Idempotency, Webhooks & Double-Entry Ledgers

#architecture #backend #fintech #microservices

A practical deep dive into handling mobile money STK pushes, preventing double-credits with distributed locks, and building robust ledger pipelines.

Handling real-money transactions in backend microservices comes with a harsh reality: networks fail, webhooks retry unpredictably, and duplicate requests will happen.

Whether you are integrating mobile money APIs (like M-Pesa Daraja or IntaSend) or card processors, standard CRUD patterns fall short. If an STK Push takes 12 seconds to resolve on a customer's handset and the webhook delivers three identical callbacks within 500ms, your system must guarantee that customer balances, creator payouts, and platform accounts remain mathematically consistent.

In this article, we’ll explore the core patterns needed to build production-grade payment pipelines: idempotency keys, distributed state handling, and immutable double-entry ledgers.

1. The Realities of Asynchronous Payment Flows

A typical mobile money STK push flow is asynchronous by design:

  1. Client Request: Initiates a payment checkout (e.g., tipping a creator or booking a ride).
  2. Push Initiation: Your backend requests an STK push prompt from the payment gateway.
  3. Gateway Response: The gateway immediately returns an acknowledgment (e.g., MerchantRequestID / CheckoutRequestID), putting the transaction in PENDING.
  4. User Action: The user inputs their PIN on their phone.
  5. Webhook Callback: The gateway fires a callback to your backend endpoint with success/failure status. 

[Client] ---> (POST /checkout) ---> [Payment Service] ---> (STK Push API) ---> [Gateway]
|                                      |
Save Status: PENDING                       Prompt Handset
|                                      |
[Client] <--- (Poll / WebSocket) <---------+                                      |
|
[Payment Service] <--------------- (Webhook Callback) <---------------------------+
Enter fullscreen mode Exit fullscreen mode

Why Naive Implementations Break

  • Webhook Duplication: Upstream payment gateways retry webhooks if your server doesn't respond with a 200 OK within their timeout window (often 3–5 seconds).
  • Out-of-Order Delivery: A failure webhook might arrive after a retry, or two concurrent callback worker threads might attempt to credit a wallet simultaneously.
  • Race Conditions: If two threads read balance $10, add $5, and write $15, you introduce financial discrepancies.

2. Enforcing Strict Idempotency

Every webhook payload includes unique identifier pairs (such as TransactionID or CheckoutRequestID). You should treat these as an Idempotency Key.

Implementing Distributed Locks with Redis

Before processing an incoming callback, acquire a short-lived atomic lock using Redis (SET key value NX PX):

func (s *PaymentService) ProcessCallback(ctx context.Context, payload WebhookPayload) error {
    lockKey := fmt.Sprintf("lock:payment:%s", payload.CheckoutRequestID)

    // Acquire lock for 10 seconds
    acquired, err := s.redis.SetNX(ctx, lockKey, "locked", 10*time.Second).Result()
    if err != nil {
        return fmt.Errorf("failed to contact redis: %w", err)
    }
    if !acquired {
        // Another thread or instance is currently processing this transaction
        return ErrConcurrentProcessing
    }
    defer s.redis.Del(ctx, lockKey)

    // Run database transaction
    return s.repo.ExecuteTransaction(ctx, func(tx *sql.Tx) error {
        // 1. Check if transaction is already marked COMPLETED/FAILED
        // 2. Apply status update
        // 3. Post journal entries
        return nil
    })
}
Enter fullscreen mode Exit fullscreen mode

If another request with the same ID arrives while the lock is active, immediately return 429 Too Many Requests or 200 OK (if verified already processed), preventing concurrent database updates.

3. The Double-Entry Ledger Pattern

A common anti-pattern in payment apps is updating a single row:

-- DANGEROUS: Do not do this for core ledger balances
UPDATE accounts SET balance = balance + 50.00 WHERE id = 'user_123';
Enter fullscreen mode Exit fullscreen mode

If a bug slips into your codebase or a write fails halfway, reconciling where funds came from or went becomes impossible.

Why Double-Entry?

Every financial event consists of two or more entries:

$$\sum \text{Debits} = \sum \text{Credits}$$

For instance, when a customer deposits $10:

  • Debit: Gateway Clearing / Cash In Transit (Asset increases)
  • Credit: Customer Wallet (Liability to user increases)

Schema Design (PostgreSQL) 

CREATE TABLE accounts (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    code VARCHAR(50) UNIQUE NOT NULL, -- e.g., 'ASSET_MPESA_CLEARING', 'LIABILITY_USER_WALLET'
    currency VARCHAR(3) DEFAULT 'USD',
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

CREATE TABLE journal_entries (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    reference_id VARCHAR(100) UNIQUE NOT NULL, -- e.g., Payment Gateway TransID
    description TEXT NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

CREATE TABLE ledger_lines (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    journal_entry_id UUID NOT NULL REFERENCES journal_entries(id) ON DELETE RESTRICT,
    account_id UUID NOT NULL REFERENCES accounts(id) ON DELETE RESTRICT,
    amount NUMERIC(14, 4) NOT NULL, -- Positive for Credit, Negative for Debit
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);

CREATE INDEX idx_ledger_lines_account ON ledger_lines(account_id);
Enter fullscreen mode Exit fullscreen mode

Balance Calculation via Views & Snapshots

To get an account balance, aggregate ledger lines:

SELECT SUM(amount) AS balance 
FROM ledger_lines 
WHERE account_id = 'your-account-uuid';
Enter fullscreen mode Exit fullscreen mode

Performance Tip: For high-throughput services, do not calculate lifetime sums on every read. Use periodic snapshot tables (e.g., daily balances) and compute only the delta since the last snapshot.

4. Handling Webhook Deadlines: Decoupling with Event Streams

Payment gateways expect webhooks to return HTTP 200 almost instantly. If your ledger involves external audit logs, notification dispatches, and third-party webhooks, running them in the request-response thread will cause gateway timeouts.

Instead, split the responsibility using an event stream (like Kafka or RabbitMQ):

[Gateway] ---> [Webhook Ingestion API]
                      |
           (1) Verify Signature & Store Raw Payload
           (2) ACK Gateway HTTP 200
           (3) Emit 'payment.received' Event
                      |
                      v
             [Message Broker / Kafka]
                      |
         +------------+------------+
         v                         v
[Ledger Service]           [Notification Service]
(Updates Balances)         (Sends SMS / Push)
Enter fullscreen mode Exit fullscreen mode

Ingestion Handler (Lightweight & Safe) 

@PostMapping("/api/v1/callbacks/mpesa")
public ResponseEntity<String> handleMpesaCallback(@RequestBody String rawPayload,
                                                  @RequestHeader("X-Signature") String signature) {
    // 1. Verify HMAC / Signature
    if (!signatureVerifier.isValid(rawPayload, signature)) {
        return ResponseEntity.status(HttpStatus.UNAUTHORIZED).body("Invalid signature");
    }

    // 2. Persist raw callback in an inbox/event store
    InboxEvent event = inboxService.saveRawEvent(rawPayload);

    // 3. Publish to message bus
    kafkaTemplate.send("payments-inbox-topic", event.getId().toString(), rawPayload);

    // 4. Return fast
    return ResponseEntity.ok("{\"ResultCode\":0,\"ResultDesc\":\"Accepted\"}");
}
Enter fullscreen mode Exit fullscreen mode

5. Checklist for Production Readiness

Before releasing your payment service to live traffic, ensure you have verified:

  • [ ] State Machine Invariants: A transaction in state COMPLETED or FAILED cannot transition to any other state.
  • [ ] Negative Amount Safeguards: Add database-level check constraints (amount > 0 or balanced sign conventions) to prevent negative debit exploits.
  • [ ] Reconciliation Jobs: Run a nightly batch job that compares gateway settlement statements against your internal ledger entries.
  • [ ] Secrets & Signature Verification: Never trust incoming callback payloads without validating cryptographic HMAC headers or API gateway IP whitelists.

Conclusion

Payment architectures forgive very little. By combining idempotent locks, an immutable double-entry ledger, and asynchronous event ingestion, your system can withstand gateway hiccups, network timeouts, and duplicate payloads with zero balance discrepancies.

Have you tackled payment reconciliation or STK Push challenges in your projects? Drop a comment below with your architectural choices or pain points!


---

### Tips for Publishing on DEV.to:
1. **Tags:** DEV.to works well with 4 specific tags (e.g., `#backend`, `#architecture`, `#go`, `#java`).
2. **Cover Image:** You can swap the Unsplash placeholder URL in the front matter with a custom banner or architecture diagram.
3. **Canonical Link:** If you also publish this on your personal blog, add `canonical_url: [https://paulowuor.me/blog/](https://paulowuor.me/blog/)...` in the front matter to boost SEO without duplicating penalties.

<ElicitationsGroup message="Next steps to customize this article:">
  <Elicitation label="Add a specific framework tutorial (e.g., Spring Boot or Go Fiber)" query="Add complete code examples in Go using Fiber and SQLx for the webhook handler and ledger transaction."/>
  <Elicitation label="Generate a social media promo thread for this post" query="Draft a 5-tweet promotional thread and a LinkedIn post summarizing this article."/>
  <Elicitation label="Add an architecture diagram in Mermaid syntax" query="Add Mermaid.js diagrams illustrating the distributed locking and double-entry flow to this article."/>
</ElicitationsGroup>
Enter fullscreen mode Exit fullscreen mode

Top comments (0)