DEV Community: Jesse Muuya

What Happens After the Fifth Retry? Escalating Stuck Payments Instead of Looping Forever

Jesse Muuya — Tue, 29 Sep 2026 12:47:51 +0000

A week ago I wrote about a payment that got stuck in PENDING, and the fix I shipped: a reconciliation job that only marks a payment settled after confirming the customer actually got what they paid for, not before. A reader, Mihai Perdum, left a comment on that post that I didn't have a good answer to at the time:

Is there a retry-count or age threshold that pages someone if fulfillment repeatedly fails structurally rather than transiently?

There wasn't. The job I'd shipped would retry a failing payment every five minutes, forever, with no distinction between "the database had a blip, try again" and "this event was deleted and will never exist again, stop trying." A payment stuck for a structural reason would loop quietly until it aged out of the sweep window and disappeared with nobody told.

This is the fix, and what testing it taught me about my own first attempt.

The gap

Here's the shape of the job from the last post:

try {
  await onPaymentSettled(payment);
  await db.payment.update({ where: { id: payment.id }, data: { status: 'SUCCESS' } });
} catch (err) {
  // reverts to PENDING, retries next tick
}

Every failure looked the same to this code: log it, leave it PENDING, try again in five minutes. That's correct for a transient failure — a timeout, a dropped connection. It's wrong for anything that will never succeed on retry: a deleted event, a sold-out ticket tier, a foreign key that no longer resolves. Those don't need patience, they need a human.

The fix: count, classify, escalate

Two additions. An attempt counter, and a classifier that distinguishes "try again" from "stop and tell someone."

} catch (err) {
  const attempts = payment.fulfilmentAttempts + 1;
  const structural = isStructuralFailure(err);
  const escalate = structural || attempts >= MAX_FULFILMENT_ATTEMPTS;

  await db.payment.update({
    where: { id: payment.id },
    data: {
      fulfilmentAttempts: attempts,
      lastFulfilmentError: String(err.message).slice(0, 500),
      ...(escalate && { needsAttention: true }),
    },
  });

  if (escalate) {
    await notifyAdmins(payment, err);
  }
}

The sweep query excludes anything flagged needsAttention, so an escalated payment stops being silently retried and instead sits somewhere a human will actually see it.

Where my first version got it wrong

The classifier's job is deciding which failures are structural. My first pass:

function isStructuralFailure(err) {
  return err.name === 'PrismaClientValidationError' || err.code === 'P2025';
}

I tested it against a real synthetic failure — a payment referencing an event that didn't exist — expecting immediate escalation. Instead:

❌ Fulfilment failed for payment OTV-TEST-RETRYCAP-A (attempt 1, transient) — reverting to PENDING for retry

Four more identical failures later:

❌ Fulfilment failed for payment OTV-TEST-RETRYCAP-A (attempt 5, transient) — ESCALATING, needs manual attention

It escalated, but only after burning through the whole retry budget on something that could never have succeeded. The actual error was a plain 404 thrown by application code, not a Prisma error at all, so my classifier didn't recognize it. Fixed:

function isStructuralFailure(err) {
  const httpStatus = typeof err.getStatus === 'function' ? err.getStatus() : 0;
  return (
    (httpStatus >= 400 && httpStatus < 500) ||
    err.name === 'PrismaClientValidationError' ||
    err.code === 'P2025' || err.code === 'P2003' || err.code === 'P2002'
  );
}

Same test, rebuilt:

❌ Fulfilment failed for payment OTV-TEST-RETRYCAP-A (attempt 1, structural) — ESCALATING, needs manual attention

One attempt instead of five. The lesson: a classifier is a claim about what your errors look like, and the only way to know if the claim is true is to throw a real one at it and watch.

Making sure the alert doesn't page twice

Webhook deliveries and retries can both re-enter the failure path after a payment is already flagged. Without a guard, an already-escalated payment would generate a fresh alert on every subsequent retry attempt — not wrong, exactly, but noisy enough that people learn to ignore it. The fix is checking whether the flag was already set before sending anything:

if (escalate && !payment.needsAttention) {
  await notifyAdmins(payment, err);
}

Tested by escalating a payment once, confirming one notification row, then sending the identical failing delivery again and confirming the notification count stayed at one. It did.

Shipping the same discipline in the sold version

I maintain a small payments engine (African Payment Gateways Engine) that includes a reconciliation script with the same fulfil-before-settle shape from the last post. Auditing it after building this fix found it had the exact gap Mihai described — no counter, no classifier, retry forever. Same fix, ported: an attempt counter and needsAttention column on the payment row, the same structural-vs-transient split, the same exclusion from the sweep once flagged.

Tested the same way — a synthetic payment referencing a nonexistent event:

❌ Reconcile failed for TEST-REF-2 (attempt 1, structural) — ESCALATING, needs manual attention: event not found

And confirmed an escalated row stops being swept while ordinary pending payments keep being checked normally:

📡 Found 0 PENDING records awaiting verification polling.

(The escalated row, still PENDING in the database, correctly excluded from that batch.)

What this doesn't solve

The counter only fires when something actually calls the fulfilment hook and it throws. If the job that's supposed to be running stops running entirely — crashes, gets undeployed, a scheduler misconfiguration — a paid-but-unfulfilled payment sits there with nothing watching it, because nothing ever entered the catch block to start counting. Closing that gap needs a second, independent check: something that looks at "how old is the oldest payment the provider says succeeded but we haven't marked settled," regardless of whether the reconciliation job ran at all. That's not built yet. It's the next thing on the list.

Thanks to Mihai for the comment that started this. If your reconciliation job treats every failure the same way, it's worth checking whether that's actually true — the fastest way I know is the same one that caught my own mistake: throw the exact error you expect at it, and read what comes back.

If you're building this yourself: African Payment Gateways Engine ships the fulfil-before-settle reconciliation pattern with this retry-cap and escalation built in. The free africa-payments-utils package covers phone normalization and webhook signature verification if that's all you need.

Idempotency Isn't Enough — What I Got Wrong About M-Pesa Reconciliation

Jesse Muuya — Tue, 22 Sep 2026 12:56:21 +0000

A few weeks ago I wrote about handling M-Pesa STK Push timeouts and webhook verification. In the comments, Justin Wilson pushed back on the reconciliation snippet I'd shared:

"The key missing safeguard is idempotency. Store transaction keys and make retries safe so delayed callbacks cannot create duplicate orders."

He was right, and it turned out to be more right than either of us knew at the time. A few days after that post, one of my own production payments got stuck in exactly the way the article described. Chasing it down didn't just confirm Justin's point, it surfaced a second bug hiding behind the first one: a naive reconciliation job can be perfectly idempotent and still leave a paying customer with nothing.

This is the writeup of what actually happened, and the fix that came out of it.

The setup

A customer initiated an M-Pesa payment through STK Push. They got the prompt on their phone, entered their PIN, and — nothing. No confirmation in the app. I checked the payment record: PENDING. I checked the logs for anything payment-related: nothing. Not a single line.

That absence of logs was the first real problem. Before I could fix anything, I had to be able to see what was happening at all.

Finding the actual failure

Adding logging at the charge-initiation point immediately explained the "no logs" mystery, and revealed the shape of the bug:

Paystack M-Pesa charge response: http=200 status=true dataStatus=pay_offline
display=Please complete authorization process on your mobile phone

The charge had initiated correctly. pay_offline is Paystack's way of saying "waiting on the customer to finish on their phone" — this is the expected state right after the STK prompt goes out. The problem was downstream: the webhook that's supposed to confirm the final result never arrived, or never got processed. The payment sat in PENDING because nothing ever told it otherwise.

This is exactly the failure mode from my original post: the callback drops, the transaction is stuck, and only a background reconciliation job can recover it.

So I built one. Almost.

The fix that wasn't quite a fix

I added a scheduled job that sweeps PENDING payments, re-verifies each one directly against Paystack, and updates the status. Structurally, it looked a lot like the snippet from my first post:

async function reconcilePendingTransactions() {
  const pending = await db.payment.findMany({ where: { status: 'PENDING' } });

  for (const tx of pending) {
    const result = await verifyWithPaystack(tx.reference);

    if (result.status === 'success') {
      await db.payment.update({
        where: { id: tx.id },
        data: { status: 'SUCCESS' },
      });
    }
  }
}

I also added the thing Justin flagged: real safeguards against duplicate processing. Ticket issuance is guarded by an "already issued for this payment?" check. Stream access grants are an idempotent upsert keyed on user + resource. Running this job twice, or on three server replicas at once, can't create two tickets for the same payment or double-grant access. On that specific point, Justin's fix was already correct and already built in.

The job ran. It found the stuck payment. It verified it with Paystack: real success, real money moved. It flipped the row to SUCCESS. And the customer still didn't get their ticket.

The bug underneath the bug

Look again at that reconciliation snippet, and at the excerpt from my original post's version too. Both do the same two things, in the same order:

Confirm the payment succeeded with the provider.
Mark the row SUCCESS.

Fulfilment — actually granting whatever the customer paid for — happens somewhere else, assumed to follow from step 2. In my production code, it did follow, almost every time. This time, it threw an error partway through (a tenant-context bug in how the job ran, unrelated to payments specifically), and by the time it failed, the payment row already said SUCCESS.

That's the failure mode neither Justin's comment nor my original post's snippet accounts for: once a payment is marked SUCCESS, nothing ever looks at it again. The reconciliation job only sweeps PENDING rows. A payment that's SUCCESS-but-unfulfilled isn't stuck in the sense the job was built to detect — it's stuck in a way that's invisible to it. The customer paid. The system believes the transaction is closed. Nobody is delivered anything, and nothing is watching for that specific gap.

Idempotent retries protect you from doing fulfilment twice. They don't protect you from marking something settled before fulfilment happens even once.

The actual fix

The fix is to invert the order: fulfil first, and only mark the payment SUCCESS once fulfilment has actually succeeded.

async function reconcilePendingTransactions(
  onPaymentSettled: (payment: Payment) => Promise<void>,
) {
  const pending = await db.payment.findMany({ where: { status: 'PENDING' } });

  for (const tx of pending) {
    const result = await verifyWithPaystack(tx.reference);
    if (result.status !== 'success') continue;

    try {
      // Grant the ticket / stream access / whatever they paid for FIRST.
      // Must be idempotent — this can run more than once for the same
      // payment, exactly as Justin described.
      await onPaymentSettled(tx);

      // Only mark SUCCESS once fulfilment has actually happened.
      await db.payment.update({
        where: { id: tx.id },
        data: { status: 'SUCCESS' },
      });
    } catch (err) {
      // Fulfilment failed. Leave the row PENDING. The next run will pick
      // it back up and try again — this is what makes the retry safe,
      // and it's why fulfilment has to be idempotent in the first place.
      console.error(`Fulfilment failed for ${tx.reference}, will retry`, err);
    }
  }
}

One line moved — the fulfilment call went from implicit ("something else handles this after") to explicit and sequenced before the status update — and it closes a gap that idempotency alone doesn't touch. The two ideas work together: idempotency makes retrying safe, and fulfil-before-settle is what guarantees a retry actually happens when something goes wrong.

I traced through both my production system and a payments package I maintain separately, and found the identical bug in both: a reconciliation job that marked things settled before confirming delivery, with a // TODO: connect this to your fulfilment logic comment standing in for something that was never actually wired up. Comments don't retry. Only code does.

What this actually looked like once it was fixed

In production, this runs as a scheduled job across three server replicas behind a load balancer, coordinated with a simple distributed lock so only one replica runs the sweep at a time:

[replica-1] Scheduled reconcile: another replica holds the lock — skipping.
[replica-2] Scheduled reconcile: checked=1 synced=1
[replica-3] Scheduled reconcile: another replica holds the lock — skipping.

And when I deliberately broke fulfilment to test the failure path:

Fulfilment failed for OTV-xxxx — reverting to PENDING for retry

The row went back to PENDING, exactly as intended, and the next tick picked it up and retried cleanly once the underlying bug was fixed. That log line is the actual proof this works — not "the tests pass," but "watch it fail safely and recover."

The short version, for anyone who just wants the rule

If your reconciliation job (or your webhook handler, for that matter) can mark something SUCCESS and fulfilment can independently fail, you have this bug, whether or not your fulfilment logic is idempotent.

Idempotent fulfilment protects you from double-delivering when a retry happens.
Fulfil-before-settle is what guarantees a retry happens at all.

You need both. I had shipped code — and sold code — with only the first one, and it took a real stuck payment to notice the second one was missing.

Thanks to Justin for the original comment. It was right, and chasing down exactly how right it was is what turned into this post.

If you're integrating Paystack, Flutterwave, or M-Pesa STK Push in Node.js and want the fixed version of this pattern (fulfilment hook required, throws until you wire it in, so it can't ship silently broken) rather than building it from scratch: African Payment Gateways Engine. The free africa-payments-utils package covers the phone-normalization and webhook-signature pieces from my first post, MIT licensed, no strings attached.

How to Handle M-Pesa STK Push Timeouts & Webhook Verification in Node.js (TypeScript)

Jesse Muuya — Mon, 14 Sep 2026 07:06:41 +0000

Integrating African payment gateways like M-Pesa, Paystack, and Flutterwave into a Node.js backend comes with three recurring failure points:

M-Pesa STK Push Timeouts: Users enter their PIN late, network requests drop, and the callback never reaches your server.

Inconsistent Phone Formatting: Accepts 0712..., 254712..., or +254712..., breaking gateway payloads.

Payload Spoofing: Accepting unvalidated webhook bodies directly into your database.

Here is how to structure a resilient, zero-hosting-overhead TypeScript solution for these edge cases.

1. Normalizing Local Phone Numbers
M-Pesa and regional gateways strictly require E.164 string formats (+254XXXXXXXXX). Hand-rolling regex during API calls leads to silent runtime crashes.

Use a dedicated normalizer function before sending your STK push payload:

TypeScript

export function normalizeKenyanPhone(phone: string): string {
  // Strip spaces, dashes, and non-numeric characters except leading +
  let cleaned = phone.replace(/[^\d+]/g, '');

  if (cleaned.startsWith('0')) {
    cleaned = `+254${cleaned.slice(1)}`;
  } else if (cleaned.startsWith('254')) {
    cleaned = `+${cleaned}`;
  } else if (!cleaned.startsWith('+254')) {
    throw new Error(`Invalid local phone number format: ${phone}`);
  }

  return cleaned;
}

2. Strict Runtime Webhook Validation (Zod)
Never trust gateway request bodies without checking payload shapes at runtime. Using Zod allows you to validate raw JSON bodies before touch points with database ORMs like Prisma.

TypeScript

// validation-shapes.ts
import { z } from 'zod';

export const PaystackWebhookSchema = z.object({
  event: z.string(),
  data: z.object({
    id: z.number(),
    status: z.enum(['success', 'failed', 'abandoned']),
    reference: z.string(),
    amount: z.number(),
    currency: z.string().default('KES'),
    metadata: z.object({
      tenantId: z.string(),
      userId: z.string().optional(),
    }).passthrough(),
    customer: z.object({
      email: z.string(),
    }),
  }),
});

export type PaystackWebhookPayload = z.infer<typeof PaystackWebhookSchema>;

3. Recovering from Dropped Webhooks
When M-Pesa callbacks fail due to network timeouts, the transaction remains stuck in a PENDING state. The fix is a non-blocking background reconciliation task running on a schedule (e.g., via node-cron or serverless cron triggers) to query the provider's query API directly:

TypeScript

// Example background polling logic
async function reconcilePendingTransactions() {
  const pendingTx = await db.transaction.findMany({ where: { status: 'PENDING' } });

  for (const tx of pendingTx) {
    const statusResponse = await queryMpesaExpressStatus(tx.checkoutRequestId);

    if (statusResponse.ResultCode === '0') {
      await db.transaction.update({
        where: { id: tx.id },
        data: { status: 'SUCCESS' },
      });
    }
  }
}

Need the Production Microservice Boilerplate?
If you don't want to build the entire infrastructure from scratch, I've packaged a production-ready, framework-agnostic TypeScript engine containing:

Raw-buffer webhook signature verification (Paystack, Flutterwave, M-Pesa).

Pre-built Prisma database schemas with multi-tenant support.

Self-healing reconciliation cron jobs for timed-out callbacks.

React/Vite PWA checkout components.

👉 Get the African Payment Gateways Engine ($19–$99)
👉 View the Lite GitHub Demo