DEV Community: scielon

Stop Retrying API Requests Blindly: How Idempotency Prevents Duplicate Operations

scielon — Sat, 14 Nov 2026 16:30:38 +0000

Stop Retrying API Requests Blindly: How Idempotency Prevents Duplicate Operations

Imagine a customer clicks the Pay Now button on your website.

Your application sends a payment request to the server. The payment provider processes it successfully, but the network connection times out before your application receives the response.

What happens next?

Your application does not know whether the payment succeeded. If it simply sends the same request again, the customer might be charged twice.

This is not just a payment problem. The same situation can occur when creating user accounts, submitting orders, sending emails, registering subscriptions, or processing background jobs.

Network failures are unavoidable. Duplicate operations, however, are often preventable.

Two important techniques help solve this problem:

Idempotency keys.
Controlled retries with exponential backoff and jitter.

Understanding these techniques is essential when building reliable APIs and distributed applications.

1. Why API Requests Sometimes Produce Duplicate Operations

A common mistake in application development is assuming that a failed HTTP request means the server did not process it.

That assumption is incorrect.

Consider this sequence:

Your application sends a POST request.
The server receives and processes it.
The database records the transaction.
The server sends a successful response.
The network connection fails before the client receives that response.
The client assumes the operation failed and sends another request.

The server may process the second request as a completely new operation.

The result could be:

Two payment transactions.
Two identical orders.
Duplicate database records.
Multiple email notifications.
Repeated subscription creation.

The important distinction is between a failed request and an unknown request outcome.

A timeout tells you that the client did not receive a response within the expected time. It does not necessarily tell you what happened on the server.

2. Understand Idempotency Before Implementing Retries

In HTTP, idempotency describes an operation whose intended effect remains the same when the same request is performed multiple times.

For example, setting a user's email address to the same value repeatedly should produce the same final state.

HTTP defines PUT and DELETE as idempotent methods, along with safe methods such as GET.

POST is not inherently idempotent.

This distinction matters because HTTP clients should not automatically retry non-idempotent requests unless they have additional guarantees that make the operation safe.

The official HTTP specification explains this behavior in RFC 9110:

https://www.rfc-editor.org/rfc/rfc9110.html#section-9.2.2

However, developers can implement idempotency for POST operations using a unique identifier.

This identifier is commonly called an Idempotency Key.

3. How Idempotency Keys Work

An idempotency key identifies one logical operation.

Instead of treating every incoming request as a completely new operation, the server checks whether it has already processed the supplied key.

For example:

POST /api/payments
Content-Type: application/json
Idempotency-Key: order-8472-payment-attempt-01

The server receives the request and checks whether the key already exists.

There are two possible outcomes.

First request:

The key does not exist.
The server processes the payment.
The result is stored alongside the key.
The server returns the response.

Repeated request:

The same key is received again.
The server recognizes the previous operation.
It returns the previously stored result instead of creating another payment.

Conceptually:

Client
  |
  | POST + Idempotency Key
  v
API Server
  |
  v
Check Idempotency Key
  |
  +---- Key exists ----> Return stored result
  |
  +---- Key is new ----> Process operation
                              |
                              v
                         Save result
                              |
                              v
                         Return response

The key is not simply a random request identifier. It must remain the same across retries of the same logical operation.

A new payment or order should receive a new key.

4. Implementing an Idempotency Key in Python

Here is a simplified example using Python and the Requests library.

First, install the dependency:

pip install requests

Then create the request:

import requests
import uuid

API_URL = "https://api.example.com/payments"

# Generate one key for this logical operation.
idempotency_key = str(uuid.uuid4())

payment_data = {
    "order_id": "8472",
    "amount": 49.99,
    "currency": "USD"
}

headers = {
    "Content-Type": "application/json",
    "Idempotency-Key": idempotency_key
}

response = requests.post(
    API_URL,
    json=payment_data,
    headers=headers,
    timeout=(3.05, 15)
)

print(response.status_code)
print(response.json())

This example demonstrates how a client can send an idempotency key.

However, generating a key on the client is only half the solution.

The server must actually support and enforce idempotency.

Also, the example assumes that the API documents support for the Idempotency-Key header. Not every API supports it.

An important implementation detail

If the request times out and you decide to retry it, reuse the original key.

Do not generate a new UUID for the retry.

For example:

# Retry the same logical operation.
response = requests.post(
    API_URL,
    json=payment_data,
    headers=headers,
    timeout=(3.05, 15)
)

The original headers dictionary contains the same key.

In production, persist the key and the operation state so that a process restart does not accidentally create a new key for an operation whose outcome is still unknown.

5. What Should the Server Store?

A reliable idempotency implementation needs more than a table containing unique keys.

A typical database record might contain:

Field	Purpose
idempotency_key	Identifies the logical operation
user_id	Identifies the requesting user
request_hash	Detects different payloads using the same key
status	Tracks processing state
response_code	Stores the original HTTP status
response_body	Stores the original result
created_at	Records when the operation started

A simplified database structure could look like this:

CREATE TABLE idempotency_records (
    id BIGSERIAL PRIMARY KEY,
    user_id BIGINT NOT NULL,
    idempotency_key VARCHAR(255) NOT NULL,
    request_hash VARCHAR(64) NOT NULL,
    status VARCHAR(30) NOT NULL,
    response_code INTEGER,
    response_body JSONB,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,

    UNIQUE (user_id, idempotency_key)
);

The unique constraint is important because two identical requests could arrive simultaneously.

Without database-level protection, both requests might check for the key before either one has saved it.

This creates a race condition.

A production implementation should:

Scope keys appropriately, such as by account and operation.
Atomically claim or create the key.
Compare the incoming request with the original request.
Prevent concurrent duplicate processing.
Store the operation result reliably.
Define what happens when processing is interrupted.

If the same key arrives with a different payload, the server should reject it rather than silently treating it as the original operation.

There is another important detail: database transactions alone cannot make an external payment provider and your local database participate in one atomic transaction.

For external side effects, use the provider's own idempotency support where available, and consider durable job processing or an outbox pattern.

6. Why Exponential Backoff Matters

Idempotency protects against duplicate operations, but it does not solve every network problem.

Sometimes a server is temporarily unavailable or overloaded.

Repeatedly sending requests without waiting can make the problem worse.

Imagine a server experiencing high traffic. Thousands of clients receive errors and immediately retry their requests.

The additional traffic can increase server load and make recovery more difficult.

This is where exponential backoff becomes useful.

Instead of retrying at fixed short intervals, the waiting time increases after each failed attempt.

A simple formula is:

delay = min(max_delay, base_delay * 2^attempt)

For example:

Attempt	Approximate delay
1	1 second
2	2 seconds
3	4 seconds
4	8 seconds
5	16 seconds

These are illustrative values, not universal production settings.

Add jitter to prevent synchronized retries

Exponential backoff alone can still cause problems.

If thousands of clients fail at the same time, they may all retry at nearly identical intervals.

Jitter adds randomness to the delay.

Here is a simple Python implementation:

import random

def calculate_retry_delay(
    attempt,
    base_delay=1,
    max_delay=30
):
    exponential_delay = min(
        max_delay,
        base_delay * (2 ** attempt)
    )

    jitter = random.uniform(
        0,
        exponential_delay * 0.5
    )

    return exponential_delay + jitter

Usage:

import time

for attempt in range(5):
    delay = calculate_retry_delay(attempt)

    print(f"Waiting {delay:.2f} seconds")
    time.sleep(delay)

This code demonstrates delay calculation only. It does not implement HTTP retries or determine whether a request is safe to repeat.

For production systems, use a retry policy that also considers HTTP status codes, request semantics, total deadlines, and server-provided retry instructions such as Retry-After.

AWS provides a useful explanation of timeouts, retries, exponential backoff, and jitter:

https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/

7. Which Requests Should You Retry?

Not every failure should trigger another request.

A sensible retry strategy distinguishes temporary failures from permanent ones.

Situation	Typical action
Connection timeout	Retry cautiously if operation is safe
HTTP 429	Respect Retry-After and rate limits
HTTP 500	Consider a limited retry
HTTP 502 or 503	Consider a limited retry
HTTP 400	Usually do not retry unchanged
HTTP 401	Correct authentication first
HTTP 403	Investigate permissions
Validation error	Correct the request

These are general guidelines. Actual behavior depends on the API's documentation and the specific error response.

For example, an HTTP 500 response does not automatically prove that a state-changing operation was not completed.

Likewise, a timeout does not prove that the server failed.

For non-idempotent operations, you need a reliable way to determine the original outcome or a server-supported idempotency mechanism before retrying.

8. Common Idempotency Mistakes

Even when developers understand the concept, implementation mistakes can undermine the entire design.

Generating a new key for every retry

This defeats the purpose of idempotency.

A retry must reuse the original key.

Using the same key for different operations

A key should identify one logical operation, not an entire user session.

Two different orders should have different keys.

Ignoring concurrent requests

A simple check-then-insert implementation is not sufficient.

Use database constraints and atomic processing to prevent simultaneous requests from performing the same operation.

Storing the key without storing the result

The server needs a reliable way to return the original operation outcome.

A key without a corresponding result or recoverable processing state is incomplete.

Retrying forever

Unlimited retries can consume resources, amplify failures, and create unexpected costs.

Use bounded retries, timeouts, and an overall execution deadline.

Forgetting key expiration

Idempotency records need a retention policy.

However, expiration should be longer than the realistic retry window for the operation. Otherwise, a delayed retry could be processed as a new operation.

The correct retention period depends on the application and API provider.

9. A Practical Checklist for Production APIs

Before deploying an API that performs important state-changing operations, review the following:

[ ] Generate a unique key for every logical operation.
[ ] Reuse the same key when retrying that operation.
[ ] Enforce uniqueness at the database level.
[ ] Validate that repeated keys use the same request payload.
[ ] Handle concurrent requests safely.
[ ] Persist operation states and results.
[ ] Set connection and response timeouts.
[ ] Use bounded exponential backoff with jitter.
[ ] Respect rate limits and Retry-After headers.
[ ] Avoid retrying permanent validation and authentication errors.
[ ] Test what happens when the server succeeds but the client loses the response.
[ ] Test concurrent duplicate requests and process restarts.

These checks are particularly important for payments, orders, subscriptions, and other operations where duplicate execution has real consequences.

Conclusion

Reliable APIs are not built by assuming that network requests will always succeed.

They are built by preparing for uncertainty.

Idempotency keys help ensure that repeating a logical operation does not create unintended duplicate effects. Exponential backoff and jitter help clients recover from temporary failures without overwhelming struggling services.

Neither technique replaces the other.

Idempotency protects the operation. Retry policies protect the communication process.

If you are building a payment system, an order-processing service, or a background job pipeline, implementing both correctly can make the difference between a fragile integration and a reliable production system.

The goal is not to eliminate network failures. It is to make sure those failures do not turn into expensive application errors.

Discussion

Have you ever experienced duplicate database records, payments, or orders because of an API timeout?

How did you solve the problem?

Do you implement idempotency keys in your applications, or do you rely on other techniques?

Share your experience in the comments.

AI assistance disclosure: This article was prepared with AI assistance. Technical explanations and code examples should be reviewed and tested in the context of the target application before production use.

More Technology Insights
For more articles covering artificial intelligence, emerging technologies, software, and practical technology guides, visit Scielon.

Stop Building AI Agents for Everything: Why Simple Automation Often Wins

scielon — Fri, 13 Nov 2026 17:56:10 +0000

Stop Building AI Agents for Everything: Why Simple Automation Often Wins

Artificial intelligence has changed the way developers think about automation. Tasks that once required complex software, multiple integrations, and hours of manual work can now be handled with AI-powered tools.

But there is a growing problem: developers are starting to use AI agents for tasks that do not actually need them.

Not every workflow needs an autonomous agent. Sometimes, a simple script, a well-designed API integration, or a traditional automation pipeline is faster, cheaper, and more reliable.

The real challenge is not building the most sophisticated AI system. It is choosing the right level of complexity for the problem.

In this article, we will explore why simpler automation can outperform complex AI agents, how to decide when an agent is necessary, and how to design workflows that are easier to maintain.

1. The Problem With Overengineering AI Automation

AI agents are impressive because they can interpret instructions, make decisions, use tools, and perform multi-step tasks.

However, these capabilities introduce additional complexity.

Consider a simple workflow:

Receive a new content idea.
Collect relevant information.
Generate a draft.
Check the output.
Send it for human review.

You could build an autonomous AI agent to handle the entire process.

But should you?

If the steps are predictable, a traditional workflow with clearly defined stages may be a better solution.

An autonomous agent introduces additional questions:

What happens if it chooses the wrong tool?
How do you prevent repeated actions?
What happens when an API request fails?
How do you verify the information it produces?
How do you control unexpected costs?

More autonomy does not automatically mean better results.

2. Understand the Difference Between Automation and AI Agents

Before choosing an architecture, understand what each approach does.

Traditional automation

Traditional automation follows predefined instructions.

For example:

Receive an input.
Validate the input.
Call an API.
Save the result.
Notify the user.

The system follows the same predictable sequence every time.

AI-powered automation

AI-powered automation adds intelligence to specific steps.

For example, an AI model might classify incoming messages, summarize documents, or extract important information.

The overall workflow can still remain structured.

AI agents

An AI agent can make decisions about what to do next, select tools, and execute multiple steps based on its current context.

This flexibility is useful when the task requires reasoning and the next action cannot always be determined in advance.

The important distinction is that an AI agent is an architectural choice, not a requirement for every AI-powered application.

3. A Practical Example: Building a Content Automation Workflow

Imagine you want to automate part of a technology blog's content production process.

Instead of asking an AI agent to manage everything independently, divide the process into smaller stages.

A practical architecture might look like this:

Content Idea
     |
     v
Input Validation
     |
     v
Research Collection
     |
     v
AI Content Generation
     |
     v
Quality Validation
     |
     v
Human Review
     |
     v
Manual Publication

Each stage has a specific responsibility.

For example:

Input validation ensures the topic is clear.
Research collection gathers supporting information.
AI generation creates the initial draft.
Quality validation checks structure and completeness.
Human review verifies accuracy and originality.
Publication remains under human control.

This approach makes problems easier to identify and fix.

If research collection fails, you do not need to rebuild the entire system.

You simply retry that stage.

4. Add Basic Validation Before Trusting AI Output

One of the biggest mistakes in AI automation is assuming that a successful API response means a successful result.

It does not.

An AI model can return a technically valid response containing incomplete, irrelevant, or inaccurate information.

For example, consider a simple Python validation function:

def validate_article(article):
    required_fields = [
        "title",
        "introduction",
        "body",
        "conclusion"
    ]

    for field in required_fields:
        if not article.get(field):
            return False, f"Missing field: {field}"

    if len(article["body"].strip()) < 500:
        return False, "Article body is too short"

    return True, "Article passed basic validation"


article = {
    "title": "Practical AI Automation",
    "introduction": "An introduction to AI workflows.",
    "body": "A detailed article body goes here...",
    "conclusion": "Choose the right architecture."
}

valid, message = validate_article(article)

print(message)

This is a basic example, not a complete quality-control system.

It checks whether essential fields exist and whether the article meets a minimum length.

More advanced validation could include:

Checking whether source URLs are accessible.
Detecting repeated sections.
Verifying required headings.
Checking whether claims have supporting sources.
Flagging suspicious or unsupported information.

Remember: structural validation cannot prove factual accuracy.

5. Design for Failure, Not Just Success

Reliable automation is not about assuming everything will work.

APIs can become unavailable. Network requests can time out. Rate limits can interrupt workflows. External services can change their behavior.

A robust workflow should account for these possibilities.

Consider these practices:

Use retries carefully.

Temporary failures can often be resolved by retrying a request.

However, repeated retries should have limits. Use exponential backoff rather than sending requests continuously.

Keep logs.

Record important events, errors, and execution times. Logs make troubleshooting significantly easier.

Avoid duplicate actions.

If a workflow is interrupted and restarted, it should not accidentally create duplicate records or repeat irreversible operations.

Use unique identifiers or idempotency mechanisms where appropriate.

Separate workflow stages.

Keep research, generation, validation, and delivery independent where possible.

This makes maintenance easier and limits the impact of individual failures.

6. When Should You Actually Use an AI Agent?

There are situations where an agent is genuinely useful.

Consider using one when:

The task requires multiple decisions based on changing information.
The next action depends on previous results.
The system needs to select between different tools dynamically.
The workflow contains unexpected scenarios that cannot easily be represented with fixed rules.
The value of autonomous reasoning justifies the additional complexity.

For example, a research assistant that searches several sources, compares findings, identifies missing information, and decides what to investigate next may benefit from an agent-based architecture.

On the other hand, automatically resizing images, sending notifications, or moving structured data between applications usually does not require an autonomous agent.

A useful rule is:

Use deterministic automation for predictable tasks, AI models for tasks requiring interpretation, and agents for tasks requiring dynamic decision-making.

7. A Simple Decision Framework

Before building an AI automation system, ask yourself these questions:

Question	Recommended approach
Are the steps always predictable?	Traditional automation
Does one step require language understanding?	AI-powered automation
Does the system need to decide what to do next?	Consider an AI agent
Are errors expensive or difficult to reverse?	Add human approval
Can the task be completed with a simple API call?	Avoid unnecessary agents

This framework helps prevent unnecessary engineering.

The objective is not to use as much AI as possible. It is to solve the problem with the smallest reliable system.

8. The Future of AI Automation Is Not Necessarily More Autonomy

As AI models become more capable, it is tempting to assume that every application should become fully autonomous.

But production systems have different requirements from experimental demonstrations.

A successful application needs predictable behavior, manageable costs, clear error handling, security controls, and measurable results.

In many cases, the strongest architecture will combine traditional software engineering with AI capabilities in carefully selected places.

Developers who understand when not to use AI may build better systems than those who try to introduce an agent into every workflow.

Conclusion

AI agents are powerful, but power without a clear purpose creates unnecessary complexity.

Start with the problem, map the workflow, identify where intelligence is genuinely needed, and choose the simplest architecture that meets your requirements.

You can always introduce more autonomy later when the workflow demonstrates a real need for it.

Build systems that are reliable first and intelligent where necessary.

Discussion

What is your approach to AI automation?

Do you prefer structured workflows with AI-powered steps, or do you believe autonomous agents should handle most tasks?

Share your experience in the comments.

AI assistance disclosure: This article was prepared with AI assistance and reviewed for clarity and technical consistency.

Windows 11 Backup and Recovery: OneDrive vs File History vs Recovery Tools

scielon — Fri, 13 Nov 2026 17:37:48 +0000

One important Windows backup detail: a Recovery Drive does not contain your personal files. It can help reinstall or recover Windows, but your documents and photos need a separate backup. File History can save versions of personal files to an external or network drive, while OneDrive can synchronize selected folders and settings. These tools are useful, but they are not interchangeable. What do you currently rely on: cloud storage, an external drive, or both? Follow each community's rules and add useful context rather than posting an unexplained link.

CTA: The complete guide compares the built-in Windows options.

Article link: https://sch2030.blogspot.com/2026/11/windows-11-backup-and-recovery-onedrive.html