DEV Community

Gabriel Dressler
Gabriel Dressler

Posted on

Idempotent Confirmations, Safe Retries: Two Fixes for Payment Infrastructure

#api #backend #infrastructure

Say you're building a payment confirmation endpoint. A client calls POST /payments/{id}/confirm, your API Gateway routes it to a backend service, and that service either confirms the payment or — if it's already confirmed, which happens constantly with retried client requests — returns nothing new: an HTTP 204 No Content. Meanwhile, confirming with the actual payment processor is slow and occasionally fails transiently, so it runs as a background task with retries.

That one endpoint hits two separate, well-documented gotchas that aren't obvious until they bite you. Here's both, with the actual fix.

Gotcha #1: your gateway can turn a valid 204 into a broken 500

If you're running KrakenD in front of that backend, here's what happens by default: the backend correctly returns 204 No Content, and KrakenD's response-composition layer — which normally merges and re-encodes backend responses before sending them to the client — treats the empty body as invalid input and returns a 500 instead. This isn't a rare edge case; it's filed as a known issue in KrakenD's own repository, and the reason is structural, not a bug: KrakenD's default encoding assumes there's a body to process, merge, or re-encode.

The fix is a specific config flag, no-op output encoding, documented in KrakenD's own docs:

{
  "endpoint": "/payments/{id}/confirm",
  "method": "POST",
  "output_encoding": "no-op",
  "backend": [
    {
      "url_pattern": "/internal/payments/{id}/confirm",
      "host": ["http://payment-service:8080"]
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

no-op tells KrakenD to stop processing the response entirely for that endpoint — body, headers, and status code are passed straight through from the backend, whatever they are. The trade-off is real: you lose response composition (merging multiple backend calls into one payload) on any endpoint using no-op, so it's the right call for a single-backend passthrough like a confirmation endpoint, and the wrong call for an aggregation endpoint that legitimately needs to combine several backend responses into one.

Gotcha #2: Dramatiq's retry defaults are not what you'd guess

The background task confirming with the processor needs retries — but not blind ones. A network timeout should retry. A processor response of "card declined" absolutely should not — retrying that isn't resilience, it's a bug that could double-charge or spam a processor with requests that will never succeed.

Dramatiq's @dramatiq.actor decorator supports max_retries, min_backoff, max_backoff, and a retry_when predicate for exactly this. What catches people off guard: the built-in defaults are min_backoff=15000 (15 seconds) and max_backoff=604800000 (7 days) — sensible for a low-priority background job, almost certainly wrong for a payment confirmation a user is waiting on. Left at the defaults, a retryable failure on attempt one won't try again for 15+ seconds, and a task stuck retrying could keep backing off for a week.

import dramatiq

def should_retry(retries_so_far: int, exception: Exception) -> bool:
    # Only retry transient failures. A declined payment or invalid
    # request is not transient — retrying it is a correctness bug,
    # not resilience.
    return isinstance(exception, (TimeoutError, ConnectionError)) and retries_so_far < 5

@dramatiq.actor(
    max_retries=5,
    min_backoff=1_000,     # 1s — the 15s default is too slow for a user-facing confirmation
    max_backoff=60_000,    # cap at 1 minute instead of the 7-day default
    retry_when=should_retry,
)
def confirm_payment_with_processor(payment_id: str) -> None:
    ...
Enter fullscreen mode Exit fullscreen mode

The retry_when predicate is the part that actually matters for correctness here, not just the backoff tuning: it's what stops Dramatiq from retrying a legitimately declined payment as if it were a flaky network call.

Why both matter together

Neither of these is a deep architectural decision — they're both small, specific config choices. But they're the kind of thing that's invisible in a demo and very visible in production: the gateway returning 500 for a perfectly valid confirmation, or a background worker silently retrying a declined charge five times over five days because nobody overrode the defaults. Payment systems don't get to treat these as edge cases — they're the normal path.

Sources

Top comments (0)