Designing Idempotent Payment and Loan Workflows: A Practical Guide to Preventing Duplicate Actions

Financial applications often depend on several systems working together: a user interface, an API, a database, and one or more external service providers.

Failures can occur at any point. A client may time out after sending a request. A user may click a button twice. A service may process a request successfully while the application never receives the response.

If the application simply retries the operation, it may create duplicate records or trigger an action more than once.

Idempotency helps prevent these problems by allowing an application to recognize repeated requests that represent the same intended operation.

This article explains a practical design for idempotent workflows, using a generic financial application as an example.

1. Understand the Difference Between a Retry and a New Action

Consider a user submitting an application or initiating a payment.

The server receives the request and processes it, but the network connection fails before the response reaches the client. The client cannot tell whether the operation succeeded.

There are two possibilities:

• The original request failed before any action was completed.
• The original request succeeded, but the response was lost.

Blindly repeating the operation can be unsafe. The server needs a reliable way to distinguish a retry of the original action from a genuinely new action.

An idempotency key provides that mechanism.

2. Generate an Idempotency Key

The client generates a unique key for a single intended operation and sends it with the request.

For example:

POST /api/applications
Idempotency-Key: 4c5d8b1e-7a2f-4c91-a2d3-example
Content-Type: application/json

The value above is illustrative; a production key should be generated using a suitable unique identifier.

The important rule is that retries of the same intended operation reuse the same key. A genuinely new operation receives a new key.

Do not treat the key as authentication. The application must still authenticate the caller and authorize access to the requested resource.

3. Store the Operation State

A simplified database table might look like this:

CREATE TABLE idempotency_records (
    id BIGSERIAL PRIMARY KEY,
    actor_id BIGINT NOT NULL,
    idempotency_key VARCHAR(255) NOT NULL,
    request_hash VARCHAR(64) NOT NULL,
    status VARCHAR(20) NOT NULL,
    response_code INTEGER,
    response_body JSONB,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    UNIQUE (actor_id, idempotency_key)
);

This is a starting point, not a complete production schema. The exact columns and retention rules depend on the application and its security requirements.

Several design choices matter:

• Scope keys to the authenticated actor or another appropriate security boundary.
• Enforce uniqueness in the database, not only in application code.
• Store a request fingerprint so the same key cannot silently be reused with different input.
• Represent processing states explicitly.
• Restrict access to stored responses, which may contain sensitive information.

The unique constraint is important because two requests can arrive simultaneously. An application-level “check, then insert” sequence without database enforcement can still allow duplicates.

4. Handle Repeated Requests Deliberately

A useful request-handling sequence is:

1. Authenticate the caller and validate the request.
2. Calculate a stable fingerprint of the relevant request data.
3. Attempt to claim the idempotency key using an atomic database operation.
4. If the key is new, create the operation record and begin processing.
5. If the key already exists, compare the stored fingerprint with the incoming request.
6. If the fingerprints differ, reject the conflicting reuse.
7. If the original operation completed, return its recorded result where appropriate.
8. If the original operation is still running, return a documented in-progress response or use a controlled waiting strategy.

The application must define what happens when an operation is in progress, fails, or reaches an uncertain state. These cases should not all be treated as permission to execute the operation again.

5. Avoid Holding a Database Transaction Across a Network Call

A common design mistake is to keep a database transaction open while waiting for an external provider.

External requests can be slow or time out. Long-running transactions may hold locks, consume database connections, and complicate recovery.

For workflows involving external systems, a safer design is often to:

• Persist the operation and its current state.
• Commit the local transaction.
• Perform the external action through a controlled worker or orchestration process.
• Record the result in a subsequent transaction.
• Reconcile uncertain outcomes before retrying potentially consequential actions.

For example, if an external provider accepted a request but the response was lost, the application should first determine whether the provider completed the action. It should not assume that a timeout means failure.

Where supported, pass a stable idempotency key to the external provider as well. If the provider does not support idempotency, use its documented status lookup or reconciliation mechanisms.

6. Use a State Machine

Explicit states make recovery easier to reason about.

A simplified workflow could use:

RECEIVED
   |
   v
PROCESSING
   |
   +------> SUCCEEDED
   |
   +------> FAILED
   |
   +------> RECONCILIATION_REQUIRED

The exact transitions depend on the business operation. For example, an uncertain external response may need to move to RECONCILIATION_REQUIRED, rather than directly to FAILED.

Define which transitions are permitted and ensure that state updates are atomic. Workers should also use leases, locking, or another concurrency-control mechanism so multiple workers do not accidentally process the same operation concurrently.

A state machine does not eliminate failures, but it makes them explicit and recoverable.

7. Test the Failure Cases

A workflow that succeeds under normal conditions is not necessarily safe under retries.

Include tests for:

• The same key submitted twice.
• Two simultaneous requests using the same key.
• The same key reused with different request data.
• A timeout after the external system has completed the action.
• A worker crashing before saving the result.
• A worker restarting while an operation is in progress.
• A database conflict during record creation.
• An external service returning a temporary error.
• A retry arriving after the original operation has completed.
• An unauthorized user attempting to retrieve another user's operation result.

Verify not only the HTTP response but also the number of business operations created and the resulting database state.

For consequential operations, the most important assertion is that retries do not produce duplicate effects.

8. Add Observability Without Exposing Sensitive Data

Logs and metrics should help engineers diagnose retries and uncertain outcomes without exposing personal or financial information.

Useful telemetry may include:

• A correlation identifier.
• A pseudonymous operation identifier.
• The current workflow state.
• Retry count and outcome.
• External provider latency and response category.
• Reconciliation queue size and processing delay.

Avoid logging authentication secrets, OTPs, full identity numbers, or unrestricted request and response bodies. Apply access controls and retention limits to operational records.

9. Document the Contract for API Consumers

Idempotency is a contract between the API and its callers. Document:

• Which endpoints support it.
• How keys should be generated and reused.
• What happens when the same key is sent with different data.
• What clients receive when an operation is still processing.
• How long keys and results are retained.
• How clients can check an operation's final status.

Retention matters: if a key expires too soon, a late retry might be treated as a new operation. Choose the retention period based on the operation's risk, retry patterns, and applicable requirements.

Conclusion

Reliable financial workflows need more than retry logic. They need a deliberate strategy for identifying repeated intent, recording operation state, coordinating external services, and recovering from uncertain outcomes.

Idempotency keys, database constraints, explicit state machines, reconciliation processes, and failure-oriented tests work together to reduce duplicate actions.

The goal is not to assume that networks and services will never fail. It is to design the system so that failures and retries are expected, observable, and safe to handle.

Editorial note: This draft was prepared with AI assistance. Before publishing, review it for technical accuracy and add an example, implementation details, or lessons from a system you have personally worked on.