A Timeout Is Not a Failed Payment

A Timeout Is Not a Failed Payment

A wallet transfer request times out. The payment service has no confirmation that the debit succeeded.

It is tempting to retry immediately. But a timeout tells us only that a response did not arrive in time. It does not tell us whether the account service received the request, is still processing it, or completed the debit and lost the response on the way back.

Those cases look identical to the caller. Treating every timeout as a failure can cause a second debit. Treating it as success can tell a customer that money moved when it did not.

A timeout is an unknown outcome, not proof of failure.

Give the transfer a stable identity

Every logical transfer needs an ID that stays the same across retries. The caller should create it once and reuse it whenever it retries the request.

type TransferRequest struct {
    TransactionID string
    AccountID     string
    Amount        int64
    Currency      string
}

A retry with the same TransactionID means “check or continue this transfer.” A retry with a new ID may look like a separate transfer to the receiving service.

The ID helps only if the service applying the debit checks it. Adding a transaction ID to a request does not, by itself, prevent duplicate processing.

Make duplicate requests safe

The account service should record which transaction IDs it has processed. When it receives a repeated request, it should return or expose the existing result instead of applying the debit again.

The duplicate check, debit and transaction record need to be coordinated. Otherwise, the service could apply the debit and then stop before recording the transaction ID. A retry might apply the debit again.

A simplified example:

func (s *Service) Debit(ctx context.Context, req TransferRequest) error {
    return s.store.WithTransaction(ctx, func(tx StoreTx) error {
        processed, err := tx.HasTransaction(ctx, req.TransactionID)
        if err != nil {
            return err
        }

        if processed {
            return nil
        }

        if err := tx.ApplyDebit(ctx, req.AccountID, req.Amount); err != nil {
            return err
        }

        return tx.RecordTransaction(ctx, req.TransactionID)
    })
}

This is illustrative, not a complete payment implementation. The storage layer must also handle concurrent copies of the same request. If two retries arrive together, they must not both pass the duplicate check and apply the debit.

Keep “unknown” separate from “failed”

Suppose the account service records the debit, but the response is lost. The caller sees a timeout, while the account service has completed the operation. Marking the transfer as failed would make the payment service’s record disagree with reality.

A workflow should distinguish a confirmed failure from an unresolved outcome. A basic state model might look like this:

const (
    Pending    = "pending"
    Succeeded  = "succeeded"
    Failed     = "failed"
    Unresolved = "unresolved"
)

The exact state names depend on the system. The important part is to avoid claiming success or failure when the result is not known. Use the original transaction ID to check the existing operation rather than creating a new transfer.

Retry with limits

Retries can help with temporary failures, but repeated attempts can add pressure to an unhealthy service. A retry policy should define which errors are retryable, how many attempts are allowed and how long to wait between attempts.

Backoff spaces out retries. Jitter can help prevent many clients from retrying at the same time. Neither prevents duplicate debits. That protection must come from idempotent handling where the debit is applied.

Every attempt should use the same request:

for attempt := 0; attempt < maxAttempts; attempt++ {
    err := accountService.Debit(ctx, request)
    if err == nil {
        return nil
    }

    if !isRetryable(err) {
        break
    }

    if err := waitWithBackoff(ctx, attempt); err != nil {
        break
    }
}

return markUnresolved(request.TransactionID)

The helper functions and retry policy depend on the service. The key detail is that each attempt uses the same request, including the same transaction ID.

Reconcile before deciding

Sometimes retries and status checks still cannot establish what happened. Reconciliation can compare the payment service’s record with the account service’s record and resolve the mismatch.

The transaction ID should connect the original request, its retries and the resulting records. Reconciliation should not blindly issue another debit. Its purpose is to determine the transaction’s state and update the records accordingly.

Test the failure path

A successful request is only one case to test. A more revealing case is when the debit succeeds but the response is lost. The caller retries with the same ID, and the account service returns or exposes the original result without applying the debit again.

Also test concurrent retries and a service failure between applying the debit and recording the transaction. These cases show whether duplicate handling is enforced or only assumed.

The central question is not “Did the request fail?” It is “Did the operation fail, or did we fail to observe its outcome?” In a payment system, that distinction can keep a timeout from becoming a second transfer.