DEV Community

Rahul Mehta
Rahul Mehta

Posted on

Designing Idempotent Payment Jobs: Preventing Double Charges Across Worker Retries

#api #architecture #backend #webdev

Designing Idempotent Payment Jobs: Preventing Double Charges Across Worker Retries

A background worker dispatches a charge request to a payment gateway. The gateway processes the transaction, debits the customer's card, and prepares the response. Just before the HTTP 200 payload reaches the application server, a transient socket timeout cuts the connection.

The worker catches a connection reset error. From the application's perspective, the operation failed. Because background processing systems like BullMQ, Sidekiq, and Celery make retries as simple as setting attempts: 3, the job re-queues. Thirty seconds later, a second worker picks up the job, sends a brand-new charge request, and bills the customer a second time.

Turning on default retry policies for jobs that execute network calls against external financial APIs creates immediate liability. Database transactions cannot roll back operations across an HTTP boundary. Preventing duplicate charges requires explicit state modeling, deterministic idempotency keys, and defensive retry logic.

The In-Flight Failure: Why ACID Guarantees Stop at the Network Boundary

In a local relational database, atomicity is guaranteed:

BEGIN;
UPDATE accounts SET balance = balance - 100 WHERE id = 1;
UPDATE accounts SET balance = balance + 100 WHERE id = 2;
COMMIT;
Enter fullscreen mode Exit fullscreen mode

If the server crashes midway, the transaction aborts and state rolls back cleanly.

When communicating with external third-party APIs (Stripe, Adyen, PayPal), this boundary disappears. An HTTP request has three distinct failure modes:

  1. Pre-flight failure: The request drops before reaching the gateway. No charge occurs.
  2. Mid-flight failure: The gateway receives the request, charges the payment method, but crashes before replying.
  3. Post-flight failure: The gateway succeeds and replies, but a network hop drops the response packet, or the client worker crashes before writing the success state to the local database.

Because modes 1, 2, and 3 often produce identical client-side errors (such as ECONNRESET, ETIMEDOUT, or 504 Gateway Timeout), the worker cannot infer what happened on the provider's side from the network exception alone.

Generating Deterministic Idempotency Keys at the Domain Layer

Most modern payment APIs support an Idempotency-Key header. If the gateway receives two requests bearing the exact same key within a specific window (usually 24 hours), it executes the charge once and returns the cached response for any subsequent calls.

A common pitfall is generating this key inside the worker handler:

// ANTI-PATTERN: Generates a new key on every retry attempt
async function processPayment(job: Job) {
  const idempotencyKey = crypto.randomUUID();
  await paymentGateway.charge({
    amount: job.data.amount,
    idempotencyKey,
  });
}
Enter fullscreen mode Exit fullscreen mode

If the worker times out and retries, crypto.randomUUID() generates a new key. The gateway treats the retry as an intentional secondary payment, rendering the gateway's duplicate protection useless.

Idempotency keys must be deterministic or pre-generated during the domain event that initiated the checkout.

Option 1: Persisted Key

Create the key when inserting the initial order or payment record into the primary database:

CREATE TABLE payments (
    id UUID PRIMARY KEY,
    order_id UUID NOT NULL REFERENCES orders(id),
    amount_cents INT NOT NULL,
    status VARCHAR(32) NOT NULL,
    idempotency_key UUID NOT NULL UNIQUE,
    provider_charge_id VARCHAR(255),
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW()
);
Enter fullscreen mode Exit fullscreen mode

Option 2: Deterministic Derivation

Derive the key deterministically using immutable business identifiers:

import { createHash } from 'node:crypto';

function generatePaymentKey(orderId: string, attemptNumber: number): string {
  return createHash('sha256')
    .update(`checkout:${orderId}:attempt:${attemptNumber}`)
    .digest('hex');
}
Enter fullscreen mode Exit fullscreen mode

If the job retries, it passes the identical idempotency key to the provider.

Modeling Payment States: Explicit Transitions

Boolean flags like is_paid: false provide insufficient context during asynchronous processing. A payment record requires explicit lifecycle states to ensure workers do not double-process records that are already in flight.

[ PENDING ] ---> [ PROCESSING ] ---> [ SUCCEEDED ]
                        |
                        +-----------> [ REQUIRES_ACTION / FAILED ]
Enter fullscreen mode Exit fullscreen mode
  • PENDING: The intent is saved. No outgoing network request has succeeded or timed out.
  • PROCESSING: A worker has claimed the record. An HTTP request is either in-flight or awaiting reconciliation.
  • SUCCEEDED: The provider confirmed the funds were captured, and the external identifier is safely stored.
  • FAILED: The card was declined or definitively rejected by the provider.

When a worker starts, it must acquire the job by transitioning the row from PENDING to PROCESSING atomically.

UPDATE payments
SET status = 'PROCESSING'
WHERE id = $1 AND status = 'PENDING'
RETURNING id;
Enter fullscreen mode Exit fullscreen mode

If the query returns zero rows, another worker is either actively handling the payment or it has already finished. The current worker should exit immediately.

Handling Unclear Timeout Responses: Verify Before Resubmission

When a network timeout occurs, the worker should not immediately throw an error and let the queue hammer the endpoint again. Instead, it must confirm the state with the gateway.

Many providers offer a retrieval endpoint where transactions can be looked up by their idempotency key or client reference ID:

  1. Catch the timeout exception.
  2. Query the provider: gateway.payments.retrieve({ idempotencyKey }).
  3. If the payment exists and succeeded, update the local database to SUCCEEDED and exit cleanly.
  4. If the provider has no record of the transaction, safe re-dispatch can proceed.
  5. If the provider also returns a 5xx or times out on lookup, mark the local payment record as REQUIRES_RECONCILIATION and defer the job to an exponential backoff queue.

Structuring the Worker Handler

Below is a complete implementation of a worker handler using Node.js, an SQL client, and an external payment gateway.

import { Job } from 'bullmq';
import { db } from './database';
import { paymentGateway, GatewayTimeoutError } from './paymentGateway';

interface PaymentJobData {
  paymentId: string;
}

export async function paymentWorkerHandler(job: Job<PaymentJobData>) {
  const { paymentId } = job.data;

  // 1. Fetch the payment record
  const payment = await db.query(
    'SELECT id, amount_cents, status, idempotency_key FROM payments WHERE id = $1',
    [paymentId]
  );

  if (!payment) {
    throw new Error(`Payment ${paymentId} not found`);
  }

  // 2. Short-circuit if already processed (Idempotent locally)
  if (payment.status === 'SUCCEEDED') {
    return { status: 'already_completed' };
  }

  // 3. Prevent duplicate concurrent execution
  if (payment.status === 'PENDING') {
    const updated = await db.query(
      `UPDATE payments 
       SET status = 'PROCESSING' 
       WHERE id = $1 AND status = 'PENDING' 
       RETURNING id`,
      [paymentId]
    );

    if (updated.rowCount === 0) {
      // Another worker claimed the record
      return { status: 'concurrent_lock_miss' };
    }
  }

  // 4. Dispatch to external gateway using the persisted idempotency key
  try {
    const result = await paymentGateway.createCharge({
      amount: payment.amount_cents,
      currency: 'usd',
      idempotencyKey: payment.idempotency_key,
    });

    // 5. Update local state to SUCCEEDED inside a local transaction
    await db.query(
      `UPDATE payments 
       SET status = 'SUCCEEDED', 
           provider_charge_id = $1, 
           updated_at = NOW() 
       WHERE id = $2`,
      [result.chargeId, paymentId]
    );

    return { status: 'success', chargeId: result.chargeId };
  } catch (error) {
    if (error instanceof GatewayTimeoutError) {
      // 6. Ambiguous state: Reconcile before blindly failing
      const existingCharge = await paymentGateway.lookupByIdempotencyKey(
        payment.idempotency_key
      );

      if (existingCharge && existingCharge.status === 'paid') {
        await db.query(
          `UPDATE payments 
           SET status = 'SUCCEEDED', 
               provider_charge_id = $1 
           WHERE id = $2`,
          [existingCharge.id, paymentId]
        );
        return { status: 'reconciled_after_timeout' };
      }
    }

    // Allow BullMQ/Sidekiq to retry using configured backoff
    throw error;
  }
}
Enter fullscreen mode Exit fullscreen mode

Safe Retry Checklist

Before turning on worker retries for any transactional integration:

  • Stable Keys: Ensure the idempotency key is tied to the persistent entity, not the individual worker lifecycle.
  • State Pre-Checks: Always check whether the job has already been fulfilled prior to issuing an API call.
  • Atomic State Transitions: Guard worker execution using conditional database updates to ensure only one worker processes an intent at any given time.
  • Structured Error Handling: Treat network timeouts as ambiguous states requiring inquiry, rather than automatic failures.

Top comments (1)

Collapse
 
dev_in_the_fog profile image
Jason Y. (dev_in_the_fog) •

Such a clean explanation of a nuanced problem. Bookmarking this for reference, thanks for sharing!