Networks are unreliable but our systems cannot be.
What is Idempotency?
Idempotency is a property of API design that ensures that making the same request multiple times produces the same result as making it once. In other words, no matter how many times an idempotent API endpoint is invoked with the same set of parameters, the outcome remains unchanged after the first successful request.
In the context of API designing, idempotency is crucial to prevent unintended side effects and ensure the predictability and reliability of the API. It allows clients to safely retry requests without causing any data duplication, overwriting, or other unwanted effects.
Idempotent API for file upload. Image by author
Idempotency of an API is determined by the changes it makes to the system, not the response it provides.
To better understand the above statement, consider this example:
- Consider an API endpoint that is designed to sign up a new user account in a web application. If this API is idempotent, it means that no matter how many times the API is called with the same input data (e.g., the same email and password), it will create the user account only once, and any subsequent invocations will have no further effect.
- The API may return a successful response (e.g., status code 200) for the first request and subsequent requests, indicating that the user account already exists, but the system state remains unchanged.
- The idempotency is evaluated based on the side effect of creating the user account, not the response message.
Real World Use Cases
Payment Processing: Idempotent APIs prevent double charging when processing payments, ensuring consistent and accurate billing.
Order Processing: Idempotent APIs in e-commerce platforms avoid duplicate orders or unintended changes to order status.
File Uploads: Idempotent APIs for file uploads prevent unnecessary duplication, ensuring files are stored only once, even during retries or network issues.
Subscription Management: Idempotent APIs handle subscription requests without creating duplicate subscriptions or unwanted changes to user preferences.
Distributed Systems: Idempotent APIs in distributed systems maintain consistency and handle failures gracefully, enabling safe retries without data inconsistencies.
How to Implement Idempotency in API Design?
Assign unique identifiers: Use UUIDs or other unique identifiers for each request to track and identify requests.
Idempotent HTTP methods: Design APIs using idempotent HTTP methods like GET, PUT, and DELETE. These methods ensure that multiple identical requests have the same effect as a single request.
Expiration time for idempotency keys: Set a reasonable expiration time for idempotency keys to ensure they are valid only for a certain period.
Response codes and headers: Utilize appropriate HTTP status codes (e.g., 200, 201, 204) and headers (e.g., ETag, Last-Modified) to indicate idempotency and successful processing.
HTTP Methods and Idempotency
Idempotent methods are those that can be safely repeated multiple times without changing the result beyond the initial operation.
The HTTP methods that are idempotent are GET, HEAD, PUT, and DELETE.
POST is not idempotent. This is because each time you make a POST request, it creates a new resource on the server, leading to a different result with each request. Subsequent POST requests will create additional resources, altering the state of the server, making it non-idempotent.
What are Idempotency Keys?
Before making an API call, the client requests a random ID from the server, which acts as the idempotency key.
The client includes this key in all future requests to the server. The server stores the key and request details in its database.
When the server receives a request, it checks if it has already processed the request using the idempotency key.
If it has, the server ignores the request. If not, it processes the request and removes the idempotency key, ensuring processing is done only once.
Example
In the example below, the server generates an idempotency key and returns it to the client in the response header (Idempotency-Key). The client must include this key in all subsequent requests. The server checks if it has already processed the request using the idempotency key and ensures exactly-once processing.
Note - This example is not intended for production use; instead, it serves to illustrate the fundamental concept of idempotent keys.
Node.js (Express)
const express = require('express');
const app = express();
const { v4: uuidv4 } = require('uuid');
const idempotencyKeys = new Set();
app.use(express.json());
app.post('/api/resource', (req, res) => {
const idempotencyKey = req.header('Idempotency-Key');
if (idempotencyKeys.has(idempotencyKey)) {
return res.status(200).json({ message: 'Request already processed' });
}
const resourceId = uuidv4();
// ... add logic to create the resource
idempotencyKeys.add(idempotencyKey);
return res.status(201).json({ resource_id: resourceId });
});
app.listen(3000, () => {
console.log('Server started on port 9000');
});
Python (Flask):
from flask import Flask, request, jsonify
import uuid
app = Flask(__name__)
idempotency_keys = set()
@app.route('/api/resource', methods=['POST'])
def create_resource():
idempotency_key = request.headers.get('Idempotency-Key')
if idempotency_key in idempotency_keys:
return jsonify({'message': 'Request already processed'}), 200
resource_id = str(uuid.uuid4())
# ... add logic to create the resource
idempotency_keys.add(idempotency_key)
return jsonify({'resource_id': resource_id}), 201
if __name__ == '__main__':
app.run()
Conclusion
Idempotent APIs play a crucial role in ensuring the reliability, consistency, and efficiency of a system.
Oldest comments (26)
Had heard about idempotency but never looked into it much.
I love all your articles!
Thank youu :D
"Networks are unreliable but our systems cannot be." is powerful haha!
Learnt a lot as usual
Glad it was helpful 🙌
This was a good read. 🔥
Thank you!
I have been doing this for twenty years now. Astonishingly, I have won not a single convert in that time.
If there is a concern for collisions, you can easily generate the UUIDs on the server or get them directly from the database. I used to generate fifty at a time with a query to PostgreSQL, then load them in the
<head>of the page as a queue. When a form was submitted, it would shift a UUID off the queue and then do a PUT instead of a post passing the UUID (I generally converted them to Base58).On the back end, the PUT would either replace the record entirely if it existed, or create a new record if it didn't, returning a 200 if updated or a 201 if created, along with the full record.
I used PATCH to update records (rather than replace), returning a 200 and the updated record.
DELETE returned a 204 and GET a 200 with the record(s).
This way GET remained nullipotent and PUT, PATCH, and DELETE were idempotent. No duplicate records, no back button nonsense. Easy peasy.
If the queue ever got down to say, ten or twenty, I'd do a query to bump it back up again. Again, this was probably overly cautious, but I like rock solid reliability.
You can also just generate the UUIDs on the client as needed. You can use version 1 if you're OK with the MAC address being encoded. Twenty years ago the quality of the JS UUID libraries was questionable. These days you can probably get away with
crypto.randomUUID()on most browsers.With PostgreSQL for example, you can simply do
INSERT ... ON CONFLICT (id) DO UPDATE ....I have never understood why there was so much push back against this method.
It's tough to convince others to adopt new methods despite their effectiveness. 😅
Thank you for sharing your experience. Really helpful.
Thanks @chasm for sharing your approach!
I want to understand how do you ensure that these UUIDs are not misused in the system? Is it by authorization? Do UUIDs have expiry then?
Thanks, Mudit. I don't really understand your question. UUIDs are unguessable. They only represent what you choose to make them represent. It's not like they are crypto – you can't "decode" them. They are just big numbers.
So your question sounds to me like "how dow you ensure that 7 and 9 are not misused in the system?" and "does 33 have an expiry?"
Those questions make no sense to me, but maybe I am misunderstanding you or we are talking about two different things. Sorry if I'm missing something.
What I was trying to ask is if there are security concerns of pre-generating UUIDs and in that case, should they have an expiry?
This pattern works well if we ignore network errors that leads to clients to retry. When there is a network problem to receive the uuid on the client, but the server have already commited, the idenpotency is violated. One alternative is expected from the client an UUID on the request, so it can be reused on retries. Other alternative is hash the request payload to use as identifier of what have already commited.
I had a doubt you mentioned that POST is not idempotent right. The example you gave about creating a new user if it doesn't exist what http method will you use for this. Will it be PUT?
As a verb POST is not idempotent. As part of your bussiness logic, however, you will ensure (or want to) that said request perfom as such. For the 'new user' example, email/phone/username are candidate keys. Meaning, you won't have two user with those same values
Well, by design (the minimum bare of good design), most Rest API are idempotent. Using POST as an example, if for the same dataset you are creating the 'same' resource twice, is because of your design being wrong and not because you need to enforce idempotency.
There are situations where enforcing idempotency is required even in well-designed APIs. For example - in scenarios where network issues or other errors can cause a request to be retried.
I believe enforcing idempotency is not about compensating for design flaws but rather about adding an extra layer of safety and consistency to API interactions - even if a request is duplicated/retried, the overall system remains in a predictable/consistent state.
Hello, I fully agree on this one. I see this at the company I work for, that without enforcing the idempotency at API level, we are now building other checks elsewhere.
I agree with @karishmashukla totally.
We use idempotency a lot at work (I work in payments).
Idempotence keys are not important but rather crucial for certain use cases like building robust payment systems. In the payment system, there is a risk of double payments due to retries, which can be avoided by idempotence keys.
I wonder how you can avoid that the client side requests a new UUID, and so making all of this still not Idempotent.
Can you explain about this.
Assume a client goes back 2 pages, change some of the data and submits again.
Would the latest info still be stored as new or as update, or is it then just skipped?
REST api by design should be stateless whereas in your examples you are making your server stateful.
Imho, this is over-engineering and is not really idempotent.
The example of creation of a new user is not suitable.
If post request containing the same user info is sent two times for user creation, the first request should create the user and send a status of 201 (assuming the user does not exist yet) and the second should return an error indicating why the request has failed, user already exist....therefore making it not idempotent.
Your implementation should not be using "idempotency keys" but rather concentrate on the type of resource being created. If the resource needs to be unique like email/username in the case of user creation, your function should cater for it and not rely on request keys.
What if the same request is sent with another request key...in your implementation, you would have end up with duplicate resource whereas focus should be on verifying where this email/username is available....or in the real world your servers should be able to scale horizontally and the second request would probably be executed by a different server
I would recommend you to read the article once again. The example of creation of a new user IS NOT explaining the concept of idempotence keys, rather it is to very simply explain the statement "Idempotency of an API is determined by the changes it makes to the system, not the response it provides." As you clearly stated if the user already exists we get an error response but the state of the system remains the same. This makes it simple for beginners to get the core idea :)
There is NO WHERE the author is asking you to use idempotence keys for user creation. Instead the author clearly mentions Real World Use Cases where idempotency will be of help
The author also clearly mentions "The code examples are just to illustrate fundamental concept of idempotent keys."
At my work place we do use idempotency and enforcing idempotency in production would require a lot more code and hence the code examples are just "examples" (already clarified by the author)
Thanks for pointing it out to me. I confused idempotency in rest api (for GET method) as opposed to the usage of idempotency keys to circumvent arising issues in case of retries.
No problem. Glad I could help :D
You're the best!)
Thanks a lot 🙌