I didn't learn about idempotency from a system-design interview.
I learned it while building one of my projects.
Everything seemed to be working exactly as expected.
A webhook arrived.
My backend processed it.
The database was updated.
The request completed successfully.
Then it happened again.
And again.
The same event was being processed more than once.
At first, I thought the webhook provider was doing something weird.
It wasn't.
My system simply wasn't designed for reality.
The Assumption That Broke Everything
When I first built the webhook flow, the mental model was pretty simple:
Webhook arrives
↓
Process webhook
↓
Update database
↓
Done
It feels reasonable.
An event happens → the provider sends it → my server processes it.
One event.
One request.
One operation.
Except distributed systems don't work that cleanly.
The actual world looks more like:
Webhook Provider
|
| POST /webhook
↓
My API
|
↓
Processing
|
↓
Database
And somewhere along the way, things can fail.
The provider might send the request successfully.
My server might process it successfully.
But the response might never make it back.
Maybe the connection times out.
Maybe the server crashes immediately after processing.
Maybe there is a transient network failure.
From my application's perspective:
"Everything worked."
From the provider's perspective:
"Did they receive it?"
And when the provider doesn't receive the expected response, it may retry.
So now:
Event #123
↓
Request #1 → processed
↓
Response lost
↓
Request #2 → processed AGAIN
That's where the fun begins.
The Duplicate Event
Imagine my webhook receives this:
{
"id": "evt_123",
"type": "something.completed"
}
The first request arrives:
evt_123
↓
Process
↓
Database update
↓
Success
Then the provider retries:
evt_123
↓
Process
↓
Database update AGAIN
↓
Success
The second request isn't a new event.
It's another delivery of the same logical event.
That distinction is the key.
ONE EVENT
|
┌─────────┼─────────┐
↓ ↓ ↓
Delivery 1 Delivery 2 Delivery 3
| | |
└─────────┼─────────┘
↓
ONE BUSINESS EFFECT
The transport layer can deliver the event multiple times.
My application needs to make sure the business operation remains correct.
That's idempotency.
So, What Actually Is Idempotency?
In simple terms:
An operation is idempotent when repeating it produces the same final result as performing it once.
Mathematically:
f(f(x)) = f(x)
For webhooks, that means:
Receive event
↓
Process event
↓
Receive same event again
↓
Don't create another side effect
The important part isn't necessarily preventing the second request.
You usually can't control that.
The important part is preventing the duplicate effect.
That's a subtle but important distinction.
The First Solution I Thought Of
Once I understood what was happening, the obvious solution was:
"I'll just store the webhook ID."
Something like:
processed_webhooks
+----------------+
| event_id |
+----------------+
| evt_123 |
| evt_456 |
| evt_789 |
+----------------+
Then:
Webhook arrives
↓
Extract event_id
↓
Have I processed this?
/ \
YES NO
| |
↓ ↓
Ignore Process
↓
Store event_id
Conceptually, that's correct.
But there's another trap hiding inside it.
The Race Condition
Imagine two copies of the same webhook arrive almost simultaneously.
Request A Request B
| |
↓ ↓
Check event_id Check event_id
| |
↓ ↓
"Not found" "Not found"
| |
↓ ↓
Process Process
Both requests checked before either one had recorded the event.
Now both execute the side effect.
evt_123
↓
Worker A → Process
↓
Worker B → Process
So this isn't enough:
if not already_processed(event_id):
process_event()
mark_as_processed(event_id)
There's a gap between:
check
and
insert
And concurrent requests can exploit that gap without doing anything malicious.
This is where the database becomes more than just storage.
Let the Database Enforce the Rule
Instead of trusting application logic alone, make the event ID unique.
For example:
CREATE TABLE webhook_events (
event_id VARCHAR(255) PRIMARY KEY,
processed_at TIMESTAMP NOT NULL
);
Now the database itself guarantees:
evt_123 → allowed
evt_123 → rejected
evt_123 → rejected
Even if multiple workers race to insert the same event ID, only one can win.
That changed how I thought about idempotency.
It's not simply:
if already_processed:
return
It's a data consistency invariant:
The same logical event must not be able to create multiple copies of the same business effect.
And invariants are often much safer when your database enforces them.
Then I Found the Next Problem
But there's a deeper issue.
Suppose I do:
1. Insert event ID
2. Process event
3. Return success
What happens here?
Insert event ID
↓
SUCCESS
↓
Process business operation
↓
💥 Server crashes
The event is already marked as processed.
But the actual operation never finished.
The provider retries.
My application checks the event:
evt_123
↓
Already exists
↓
Skip
Now I've prevented duplication...
but I've also prevented recovery.
That's when idempotency stopped looking like a simple duplicate-check problem.
It became a failure-handling problem.
Idempotency Is About Failure
This is probably the biggest lesson I took away from the experience.
When everything works:
Webhook
↓
Process
↓
Database
↓
Success
Almost any implementation looks correct.
The interesting cases are:
What if the request arrives twice?
What if two copies arrive simultaneously?
What if the server crashes halfway through?
What if the database succeeds but the response fails?
What if the external API succeeds but my server crashes before recording it?
What if the worker retries?
These aren't weird edge cases.
They're normal failure modes in distributed systems.
And webhooks make them particularly obvious because retries are expected behavior.
The Mental Model I Use Now
I no longer think about a webhook as:
request → function → response
I think about it as:
event → state transition
The event has an identity.
The system has a state.
And processing the same event again shouldn't produce an invalid state.
For example:
RECEIVED
↓
PROCESSING
↓
COMPLETED
If something fails:
RECEIVED
↓
PROCESSING
↓
FAILED
↓
RETRY
↓
PROCESSING
↓
COMPLETED
Now the system has a way to reason about recovery instead of simply hoping the request succeeds.
And This Doesn't Stop at Webhooks
This is where the lesson became much bigger than the original bug.
The same problem exists with:
- Webhooks
- Message queues
- Background workers
- Scheduled jobs
- Serverless functions
- Payment processing
- Event-driven architectures
- Retry mechanisms
- Distributed APIs
Anywhere something can be retried, you need to ask:
What happens if this operation runs twice?
Because eventually, it probably will.
The Real Meaning of "Exactly Once"
You'll sometimes hear people talk about exactly-once processing.
It sounds like the perfect solution:
Event → Process exactly once
But in real distributed systems, that's an extremely strong guarantee.
Your application may not be able to control how many times a message or webhook is delivered.
What it can control is whether repeated delivery produces repeated side effects.
So a much more useful design goal is:
At-least-once delivery
+
Idempotent processing
=
Correct business result
The event might arrive three times.
The system should still produce the result intended by one logical event.
That's a much more practical way to think about reliability.
Designing an Idempotent Webhook
If I were designing the webhook from scratch today, I'd start with these questions.
1. What uniquely identifies the event?
Prefer the provider's event ID when available.
evt_123
Not:
timestamp + user_id + event_type
unless you have a very specific reason to construct your own key.
2. Can the database enforce uniqueness?
Use something like:
event_id UNIQUE
or a primary key.
Don't rely solely on:
if not exists:
because concurrent workers exist.
3. What happens if processing fails halfway through?
You need a recovery strategy.
Possible approaches include:
transactions
processing states
retry queues
dead-letter queues
outbox patterns
The right choice depends on the system.
4. What about external side effects?
This one is easy to overlook.
Your database might be transactional.
The external API you're calling probably isn't part of that transaction.
If you call another service, ask:
Can I safely retry this request?
If that service supports idempotency keys, use them.
Now you have:
Webhook event ID
↓
Your idempotency key
↓
Downstream API idempotency key
The guarantee propagates through the system.
5. What happens when the worker crashes?
Assume it will.
Because eventually it will.
Design your processing state so that the system can distinguish between:
completed
failed
still processing
stuck
retryable
instead of treating everything as simply:
processed = true
What I Would Tell Myself Before Building It
If I could go back to the beginning of that project, I'd ask one question before writing the webhook handler:
"What happens if I receive this exact event twice?"
Then I'd ask:
"What happens if I receive it twice at exactly the same time?"
And finally:
"What happens if my server crashes halfway through?"
Those three questions expose a surprising amount of architectural weakness.
The Bug Wasn't Really the Bug
Looking back, the duplicate webhook wasn't the most interesting part.
The interesting part was realizing that my original mental model was wrong.
I was thinking:
Request
↓
Code
↓
Response
Production forced me to think:
Event
↓
Network
↓
Retries
↓
Concurrency
↓
Partial failure
↓
State
↓
Recovery
That's a very different way of thinking about backend systems.
And that's probably why idempotency is one of those concepts that feels trivial when you first hear the definition...
until you have to build it.
My Takeaway
Retries aren't bugs.
They're a normal part of distributed systems.
The mistake is designing your application as if retries won't happen.
A webhook provider can send the same event again.
A queue can deliver the same message again.
A worker can execute the same job again.
An API client can retry the same request again.
Your job isn't always to stop those things from happening.
Your job is to make sure repeating them doesn't corrupt the system.
That's idempotency.
And for me, it wasn't something I really understood because I read the definition.
I understood it because one of my projects broke my assumptions.
Sometimes production is a surprisingly effective teacher.
Top comments (0)