Stop Retrying API Requests Blindly: How Idempotency Prevents Duplicate Operations

Stop Retrying API Requests Blindly: How Idempotency Prevents Duplicate Operations

Imagine a customer clicks the Pay Now button on your website.

Your application sends a payment request to the server. The payment provider processes it successfully, but the network connection times out before your application receives the response.

What happens next?

Your application does not know whether the payment succeeded. If it simply sends the same request again, the customer might be charged twice.

This is not just a payment problem. The same situation can occur when creating user accounts, submitting orders, sending emails, registering subscriptions, or processing background jobs.

Network failures are unavoidable. Duplicate operations, however, are often preventable.

Two important techniques help solve this problem:

• Idempotency keys.
• Controlled retries with exponential backoff and jitter.

Understanding these techniques is essential when building reliable APIs and distributed applications.

1. Why API Requests Sometimes Produce Duplicate Operations

A common mistake in application development is assuming that a failed HTTP request means the server did not process it.

That assumption is incorrect.

Consider this sequence:

1. Your application sends a POST request.
2. The server receives and processes it.
3. The database records the transaction.
4. The server sends a successful response.
5. The network connection fails before the client receives that response.
6. The client assumes the operation failed and sends another request.

The server may process the second request as a completely new operation.

The result could be:

• Two payment transactions.
• Two identical orders.
• Duplicate database records.
• Multiple email notifications.
• Repeated subscription creation.

The important distinction is between a failed request and an unknown request outcome.

A timeout tells you that the client did not receive a response within the expected time. It does not necessarily tell you what happened on the server.

2. Understand Idempotency Before Implementing Retries

In HTTP, idempotency describes an operation whose intended effect remains the same when the same request is performed multiple times.

For example, setting a user's email address to the same value repeatedly should produce the same final state.

HTTP defines PUT and DELETE as idempotent methods, along with safe methods such as GET.

POST is not inherently idempotent.

This distinction matters because HTTP clients should not automatically retry non-idempotent requests unless they have additional guarantees that make the operation safe.

The official HTTP specification explains this behavior in RFC 9110:

https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2

However, developers can implement idempotency for POST operations using a unique identifier.

This identifier is commonly called an Idempotency Key.

3. How Idempotency Keys Work

An idempotency key identifies one logical operation.

Instead of treating every incoming request as a completely new operation, the server checks whether it has already processed the supplied key.

For example:

POST /api/payments
Content-Type: application/json
Idempotency-Key: order-8472-payment-attempt-01

The server receives the request and checks whether the key already exists.

There are two possible outcomes.

First request:

• The key does not exist.
• The server processes the payment.
• The result is stored alongside the key.
• The server returns the response.

Repeated request:

• The same key is received again.
• The server recognizes the previous operation.
• It returns the previously stored result instead of creating another payment.

Conceptually:

Client
  |
  | POST + Idempotency Key
  v
API Server
  |
  v
Check Idempotency Key
  |
  +---- Key exists ----> Return stored result
  |
  +---- Key is new ----> Process operation
                              |
                              v
                         Save result
                              |
                              v
                         Return response

The key is not simply a random request identifier. It must remain the same across retries of the same logical operation.

A new payment or order should receive a new key.

4. Implementing an Idempotency Key in Python

Here is a simplified example using Python and the Requests library.

First, install the dependency:

pip install requests

Then create the request:

import requests
import uuid

API_URL = "https://api.example.com/payments"

# Generate one key for this logical operation.
idempotency_key = str(uuid.uuid4())

payment_data = {
    "order_id": "8472",
    "amount": 49.99,
    "currency": "USD"
}

headers = {
    "Content-Type": "application/json",
    "Idempotency-Key": idempotency_key
}

response = requests.post(
    API_URL,
    json=payment_data,
    headers=headers,
    timeout=(3.05, 15)
)

print(response.status_code)
print(response.json())

This example demonstrates how a client can send an idempotency key.

However, generating a key on the client is only half the solution.

The server must actually support and enforce idempotency.

Also, the example assumes that the API documents support for the Idempotency-Key header. Not every API supports it.

An important implementation detail

If the request times out and you decide to retry it, reuse the original key.

Do not generate a new UUID for the retry.

For example:

# Retry the same logical operation.
response = requests.post(
    API_URL,
    json=payment_data,
    headers=headers,
    timeout=(3.05, 15)
)

The original headers dictionary contains the same key.

In production, persist the key and the operation state so that a process restart does not accidentally create a new key for an operation whose outcome is still unknown.

5. What Should the Server Store?

A reliable idempotency implementation needs more than a table containing unique keys.

A typical database record might contain:

Field	Purpose
idempotency_key	Identifies the logical operation
user_id	Identifies the requesting user
request_hash	Detects different payloads using the same key
status	Tracks processing state
response_code	Stores the original HTTP status
response_body	Stores the original result
created_at	Records when the operation started

A simplified database structure could look like this:

CREATE TABLE idempotency_records (
    id BIGSERIAL PRIMARY KEY,
    user_id BIGINT NOT NULL,
    idempotency_key VARCHAR(255) NOT NULL,
    request_hash VARCHAR(64) NOT NULL,
    status VARCHAR(30) NOT NULL,
    response_code INTEGER,
    response_body JSONB,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

    UNIQUE (user_id, idempotency_key)
);

The unique constraint is important because two identical requests could arrive simultaneously.

Without database-level protection, both requests might check for the key before either one has saved it.

This creates a race condition.

A production implementation should:

1. Scope keys appropriately, such as by account and operation.
2. Atomically claim or create the key.
3. Compare the incoming request with the original request.
4. Prevent concurrent duplicate processing.
5. Store the operation result reliably.
6. Define what happens when processing is interrupted.

If the same key arrives with a different payload, the server should reject it rather than silently treating it as the original operation.

There is another important detail: database transactions alone cannot make an external payment provider and your local database participate in one atomic transaction.

For external side effects, use the provider's own idempotency support where available, and consider durable job processing or an outbox pattern.

6. Why Exponential Backoff Matters

Idempotency protects against duplicate operations, but it does not solve every network problem.

Sometimes a server is temporarily unavailable or overloaded.

Repeatedly sending requests without waiting can make the problem worse.

Imagine a server experiencing high traffic. Thousands of clients receive errors and immediately retry their requests.

The additional traffic can increase server load and make recovery more difficult.

This is where exponential backoff becomes useful.

Instead of retrying at fixed short intervals, the waiting time increases after each failed attempt.

A simple formula is:

delay = min(max_delay, base_delay * 2^attempt)

For example:

Attempt	Approximate delay
1	1 second
2	2 seconds
3	4 seconds
4	8 seconds
5	16 seconds

These are illustrative values, not universal production settings.

Add jitter to prevent synchronized retries

Exponential backoff alone can still cause problems.

If thousands of clients fail at the same time, they may all retry at nearly identical intervals.

Jitter adds randomness to the delay.

Here is a simple Python implementation:

import random

def calculate_retry_delay(
    attempt,
    base_delay=1,
    max_delay=30
):
    exponential_delay = min(
        max_delay,
        base_delay * (2 ** attempt)
    )

    jitter = random.uniform(
        0,
        exponential_delay * 0.5
    )

    return exponential_delay + jitter

Usage:

import time

for attempt in range(5):
    delay = calculate_retry_delay(attempt)

    print(f"Waiting {delay:.2f} seconds")
    time.sleep(delay)

This code demonstrates delay calculation only. It does not implement HTTP retries or determine whether a request is safe to repeat.

For production systems, use a retry policy that also considers HTTP status codes, request semantics, total deadlines, and server-provided retry instructions such as Retry-After.

AWS provides a useful explanation of timeouts, retries, exponential backoff, and jitter:

https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/

7. Which Requests Should You Retry?

Not every failure should trigger another request.

A sensible retry strategy distinguishes temporary failures from permanent ones.

Situation	Typical action
Connection timeout	Retry cautiously if operation is safe
HTTP 429	Respect Retry-After and rate limits
HTTP 500	Consider a limited retry
HTTP 502 or 503	Consider a limited retry
HTTP 400	Usually do not retry unchanged
HTTP 401	Correct authentication first
HTTP 403	Investigate permissions
Validation error	Correct the request

These are general guidelines. Actual behavior depends on the API's documentation and the specific error response.

For example, an HTTP 500 response does not automatically prove that a state-changing operation was not completed.

Likewise, a timeout does not prove that the server failed.

For non-idempotent operations, you need a reliable way to determine the original outcome or a server-supported idempotency mechanism before retrying.

8. Common Idempotency Mistakes

Even when developers understand the concept, implementation mistakes can undermine the entire design.

Generating a new key for every retry

This defeats the purpose of idempotency.

A retry must reuse the original key.

Using the same key for different operations

A key should identify one logical operation, not an entire user session.

Two different orders should have different keys.

Ignoring concurrent requests

A simple check-then-insert implementation is not sufficient.

Use database constraints and atomic processing to prevent simultaneous requests from performing the same operation.

Storing the key without storing the result

The server needs a reliable way to return the original operation outcome.

A key without a corresponding result or recoverable processing state is incomplete.

Retrying forever

Unlimited retries can consume resources, amplify failures, and create unexpected costs.

Use bounded retries, timeouts, and an overall execution deadline.

Forgetting key expiration

Idempotency records need a retention policy.

However, expiration should be longer than the realistic retry window for the operation. Otherwise, a delayed retry could be processed as a new operation.

The correct retention period depends on the application and API provider.

9. A Practical Checklist for Production APIs

Before deploying an API that performs important state-changing operations, review the following:

• [ ] Generate a unique key for every logical operation.
• [ ] Reuse the same key when retrying that operation.
• [ ] Enforce uniqueness at the database level.
• [ ] Validate that repeated keys use the same request payload.
• [ ] Handle concurrent requests safely.
• [ ] Persist operation states and results.
• [ ] Set connection and response timeouts.
• [ ] Use bounded exponential backoff with jitter.
• [ ] Respect rate limits and Retry-After headers.
• [ ] Avoid retrying permanent validation and authentication errors.
• [ ] Test what happens when the server succeeds but the client loses the response.
• [ ] Test concurrent duplicate requests and process restarts.

These checks are particularly important for payments, orders, subscriptions, and other operations where duplicate execution has real consequences.

Conclusion

Reliable APIs are not built by assuming that network requests will always succeed.

They are built by preparing for uncertainty.

Idempotency keys help ensure that repeating a logical operation does not create unintended duplicate effects. Exponential backoff and jitter help clients recover from temporary failures without overwhelming struggling services.

Neither technique replaces the other.

Idempotency protects the operation. Retry policies protect the communication process.

If you are building a payment system, an order-processing service, or a background job pipeline, implementing both correctly can make the difference between a fragile integration and a reliable production system.

The goal is not to eliminate network failures. It is to make sure those failures do not turn into expensive application errors.

Discussion

Have you ever experienced duplicate database records, payments, or orders because of an API timeout?

How did you solve the problem?

Do you implement idempotency keys in your applications, or do you rely on other techniques?

Share your experience in the comments.

---

AI assistance disclosure: This article was prepared with AI assistance. Technical explanations and code examples should be reviewed and tested in the context of the target application before production use.

---

More Technology Insights
For more articles covering artificial intelligence, emerging technologies, software, and practical technology guides, visit Scielon.

Comments

[于侃]

The key is the whole design, and the failure is usually where it comes from. If the client generates a fresh key per attempt, retries are duplicates by construction - three attempts, three side effects. If the key is derived from the intent (resource id, operation, canonical payload) and stored alongside the response, the retry becomes a cache hit and the duplicate is impossible rather than merely unlikely.

Worth stating the limit too: idempotency keys prevent duplicates, not reordering. Two different keys arriving out of order still apply out of order, so anything order-sensitive needs a version comparison as well - not just a dedupe table.